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1. Technical Overview of SITES 2006 



Tjeerd Plomp 
Willem Johan Pelgrum 
Ralph Carstens 



1.1 Overview 

The International Association for the Evaluation of Educational Achievement (lEA) has 
been conducting comparative studies for 50 years. SITES 2006 is the fifth wave of surveys 
related to information and communication technology (ICT), a wave that lEA started 
with its Computers in Education Study (two studies with data collection in 1989 and 
1992)^ followed by the Second Information Technology in Education Studies (SITES); 
comprising SITES Module 1 (SITES-Ml) and SITES Module 2 (SITES-M2). 

SITES 2006 is an ambitious and demanding study involving complex procedures 
for drawing samples^ collecting data; and analyzing and reporting findings. The technical 
implementation of SITES 2006 resembles; to a large extent; the procedures used in lEAs 
past and present student achievement studies; such as TIMSS and PIRLS. However; 
the adult target populations; for instance; imply certain operational consequences and 
necessities. The SITES 2006 technical report therefore provides full documentation of 
the methodological and analytical implementation of the study which produced the 
international database underlying the international report (LaW; Pelgrum; & Plomp; 

2008) . To ensure that researchers can work effectively with the study’s findings and data 
as well as confidently replicate the procedures used to produce the international report; 
the technical report is complemented by the SITES 2006 user guide (Brese & CarstenS; 

2009) ; which describes the structure; content; and proposed usage of the international 
database. The combination of all three publications consequently allows analysts to 
confidently replicate procedures and to accurately undertake new analyses in areas of 
special interest. 

SITES 2006 has been managed by a consortium consisting (at the international 
level) of the University of Twente in Enschede; the Netherlands; the University of Hong 
Kong; the lEA Data Processing and Research Center (DPC) in Hamburg; Germany 
and the lEA Secretariat in Amsterdam; the Netherlands. The international coordinating 
center (ICC) was located at the University of Twente. A steering committee and a 
sampling referee assisted this consortium. At the national level; the daily coordination of 
the study has been carried out by national research coordinators (NRCs); data managers; 
and their national center research staff. Appendix A gives the names and addresses of the 
key personnel involved in each of these functions. 

The remainder of this chapter contains an overview of the main characteristics of 
SITES 2006. These are further elaborated in the subsequent chapters with respect to the 
technical; methodological; and analytical implementation of the study. 
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1.2 Background and Aims 

Before lEA embarked on the SITES program in the late 1990s, it had already conducted 
two surveys on the use of computers in education. The first — Computers in Education 
(CompEd) — collected school-level data in 1989 in 22 education systems (18 countries 
and four provinces or regions) and was reported by Pelgrum and Plomp ( 1991, 1993). 
The second CompEd study, with data collection in 1992, had a longitudinal component 
to allow for study of the rapid developments in this area. It also included a student 
component, the first international Functional Information Technology (FIT) test, as 
well as questionnaires for students and teachers (Pelgrum, Janssen Reinen, & Plomp, 
1993). 

The SITES program, consisting of several modules, aims to foster our 
understanding of how ICT affects the way students learn in schools. The first module 
of SITES (SITES-Ml), which was a school-based survey conducted in 1997, looked at 
the readiness of schools to integrate use of ICT in teaching and learning (Pelgrum & 
Anderson, 2001). This module was followed in 2001 by SITES-M2, which concerned 
174 case studies of innovative pedagogical practices. The study examined and compared 
the characteristics of innovative pedagogical practices that employed technology as well 
as the factors associated with such innovations in different countries (Kozma, 2003). 
SITES 2006, the third project in this series, sought to identify what pedagogical practices 
teachers and schools of different education systems were applying and how they were 
using ICT in these practices. SITES 2006, designed as a survey of representative samples 
of schools (providing Grade 8 education) and of teachers of mathematics and science, 
focused on the following research questions: 

• What are the pedagogical practices adopted in schools and how is ICT used in 
them? 

• What and how is ICT used in specific situations where ICT has been used relatively 
extensively within the pedagogical practice? 

• What teacher, school, community, and system factors are associated with different 
pedagogical approaches and ICT use, and can an explanatory model be identified? 

• What system factors are associated with different pedagogical approaches and ICT 
use? 

The study aimed to produce (i) international comparisons of various indicators, 
(ii) ICT in education policy recommendations, and (iii) in-depth analysis of how ICT 
influences teaching and learning processes. 

1.3 Conceptual Framework Overview 

The focus of SITES 2006 is on what happens in the classroom and how ICT is used in it. 
Consistent with the conceptual frameworks adopted in the previous two SITES studies 
(see Kozma, 2003; Pelgrum & Anderson, 2001), SITES 2006 takes the view that ICT- 
using pedagogical practices are part of the overall pedagogical practices of the teacher. 
For teachers, the reasons why and the ways in which they use ICT in the classroom are 
underpinned by their overall pedagogical vision and competence. However, pedagogical 
practices are not determined solely by the characteristics of the teachers, such as their 
academic qualifications and ICT-competence, but also by school- and system-level 
factors. While we can expect students’ learning outcomes to be influenced by the 
pedagogical practices they experience, we need to acknowledge that the outcomes 
(whether perceived or actual) influence teachers’ subsequent pedagogical decisions. 
This is because teacher-, school-, and system-level factors often have to change — or be 
changed — to accommodate the expected or actual impact of pedagogical practices on 
students. 
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Figure 1.1 presents the overall conceptual framework for the study. The concepts 
used in SITES 2006 relate to the four research questions for the study (see Section 1.2). 
To address these concepts, several suh-questions, deriving from different sources, were 
developed. These included: 

• Policy issues and concerns 

• Findings from S1TES-M2 

• Recent national surveys and other research outcomes (see Chapters 3 and 4). 

The study developed four questionnaires to collect information relevant to these 
concepts at system, school, and teacher level. Indicators for each of the concepts were 
thus operationalized in the questionnaire items. 

Although this chapter provides a hroad overview of the SITES 2006 framework, 
the respective chapters of the report describe in greater detail the conceptual elaboration, 
key reference literature, and the development process for each of these questionnaires. 
Thus: 

• System level — national context questionnaire (Chapter 2) 

• School level — principal and technical questionnaire (Chapter 3) 

• Teacher level — teacher questionnaire ( Chapter 4) . 

Figure 1.1: Overall Conceptual Framework for SITES 2006 
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1.4 Participating Education Systems 

In total, 22 education systems from around the world participated in SITES 2006 (see 
Table l.l). Of these, 15 had participated in SITES-Ml. 

Table 1.1 : Education Systems Participating in SITES-Ml and SITES 2006 



Education System 


SITES-Ml Participation 


SiTES 2006 Participation 


Alberta, Canada 






Catalonia, Spain 






Chile 






Chinese Taipei 






Denmark 






Estonia 






Finland 






France 






Hong Kong SAR 






Israel 






Italy 






Japan 






Lithuania 






Moscow, Russian Federation 






Norway 






Ontario, Canada 






Russian Federation 






Singapore 






Slovak Republic 






Slovenia 






South Africa 






Thailand 







1.5 Target Populations and Sample 

SITES 2006 targeted a population of schools offering education to students at the SITES 
target grade, namely Grade 8, as well as mathematics and science teachers who were 
teaching this target grade within these schools. By default, a stratified probability sample 
of 400 schools (providing Grade 8 education) and two mathematics and two science 
teachers per participating school in each participating education system was selected. 

Ghapter 6 provides the technical details of the sampling design as well as 
population definitions. Ghapter 10 gives information about the realized samples, 
sampling weights, and participation rates. Various appendices on sample allocation and 
stratification complement both chapters. 
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1.6 Instruments, Translation, and Verification 

In addition to the questionnaire designed to collect national context information from 
the NRCs, the study included three other questionnaires. These were administered 
respectively to (i) school principals, (ii) school-based ICT coordinators, and (iii) 
mathematics and science teachers. These instruments were made available in international 
English to the NRCs, who were then responsible for translating them into their national 
language(s). 

The lEA Secretariat organized a translation verification program that required 
professional translators to check the national versions against the English version (see 
Chapter 5 for details), both for the field trial phase as well as for the main study. This 
process was followed by a lay-out verification (again see Chapter 5), which was intended 
to check for any deviations from the international questionnaire layout and visual 
appearance. 

1-7 Survey Operations and Data Collection 

In 2005, the participating education systems adapted and translated the instruments, 
which were field-trialed in September and October 2005. The main data collection 
took place from February to May 2006 in the northern hemisphere and from August to 
November 2006 in the southern hemisphere. 

The guidelines for these operations were documented in detail in manuals 
developed by the lEA Data Processing and Research Center (see Chapter 7). A specific 
feature of SITES 2006 is that it is the first lEA study to apply online data collection (OD C) 
as an international option, although countries and schools can use paper-and-pencil data 
collection as well. The ODC methodology worked well during the field trial (Brecko & 
Carstens, 2007; Carstens, Brese, & Brecko, 2007 ) and most of the participating countries 
elected to use it as the principal method of data collection during the main study. Chapter 
8 presents the details. 

Because a number of education systems experienced substantial problems with 
securing high levels of school and teacher participation, extended data collection periods 
were granted to nine systems. This process involved close consultation between the 
respective NRCs and the ICC. In these systems, data were collected from schools and 
teachers during the first few weeks of the following school year, but these respondents 
were asked to answer the questionnaire for the situation relating to the previous school 
year. As mentioned in Section 7.9, there were no indications that this extension option 
resulted in substantial response bias. 

1.8 Data Processing, Analysis, and Reporting 

After the data collection, the NRCs submitted data files (conforming to the international 
record layout) to the IE A DPC, where an intensive data cleaning took place. Chapter 9 
describes this process in detail. 

Next, the statistics, analysis, tables, and figures for the SITES 2006 international 
report were produced, with the authors taking into account the hierarchical structure of 
both the sample and the data. During this phase, tables and figures were generated and 
the calculation rules for composites were defined. As this was a phase prone to errors, the 
NRCs checked the outcomes several times over against their knowledge of the database 
and the national context. Extensive checks of the data analysis scripts took place in 
parallel under the auspices of the lEA DPC s Research and Analysis Unit. 

The data analysis consisted of a descriptive part and an analytical part. For the 
description, tables and figures were created for single as well as composite items. The 
item-level statistics mainly concerned percentages and mean values that were estimated 
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using the sampling weights added to the databases. The jackknife procedure was used to 
calculate standard errors (see Chapter 10); factor and reliahility analyses were conducted 
for the composite indicators (see Chapter ll). Chapter 12 describes the standards and 
rules for reporting. Initial analyses were run within as well as between levels in order to 
address the third research question (see Section 1.2). 

The international report for SITES 2006 (Law et al.^ 2008) presents the outcomes 
of the descriptive and analytical parts. Descriptive information regarding the national 
context, the school level, and the teacher level is followed by the outcomes of initial 
explorations using correlation and multilevel analyses. The final part of the report offers 
reflections and policy-relevant recommendations. 
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2. National Context Questionnaire 
Development 

Ronald E. Anderson 
Tjeerd Plomp 



2.1 Overview 

National or country aspects sometimes account for elements of the education system 
and educational processes. The national context questionnaire (NCQ) was designed 
and administered to all national research coordinators (NRCs) in order to explore these 
possibilities. 

This chapter begins by describing the structure and content of the questionnaire. 
This section is followed by a description of the data collection^ codings indicator 
development, and other items pertaining to the analysis. 

2.2 Questionnaire and Analysis Framework 

The NCQ_addressed four principal topics: 

• Education system structure and responsibilities 

• Teacher preparation and pedagogy 

• Recent changes in ICT and pedagogical policies 

• System-wide policies and practices relating to the use of ICT. 

Table 2.1 lists the final questionnaire items divided into these four groups. The 
questionnaire includes both open-ended and multiple- choice questions. 

Given the emphasis of the SITES 2006 survey and its conceptual framework, 
the NCQ_ focused on system-related contextual variables, which were categorized into 
four spheres: (i) demographics, (ii) educational structure, (iii) pedagogy, and (iv) ICT. 
These variables were conceived as attributes of each education system or its associated 
regions (countries or provinces). Pedagogy and ICT in learning are, of course, processes 
occurring primarily at the classroom level. However, we can consider the general trends 
or patterns associated with these processes as characteristics of the overall system. 

In addition to the questionnaire data, the analysis included demographic data 
obtained largely from the United Nations Development Programme or UNDP (2006). 
Other data sources included the Canadian Council on Social Development (2006), the 
Ministry of Education Taiwan (2007), and the UNESCO Institute for Statistics (2004). 
Most of the demographic statistics obtained were for the countries or provinces in the 
year 2004; however, a few were for the year 2003. These were the most recent reliable 
statistics that could be obtained in 2007. While the analysis utilized over 100 variables, 
the number of cases was only 22 (i.e., the number of education systems participating in 
SITES 2006). The analysis in the international report is therefore constrained to a single 
level — the education system level. 
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Table 2.1 : Content of the SITES National Context Questionnaire (NCQ) 



Question 


Content Number 


Description 


A. Education System Structure and Other Characteristics 


1 Centralization 


Government level at which policies on structure (overall organization) made 


2 


Centralization 


Government level at which examinations implemented 


3 


Centralization 


Government level where teacher certification requirements set 


4 


Centralization 


Government level ot main tunding source tor public schooling 


5 


Centralization 


Share ot schools at target grade tunded by state or regional level 


6 


Centralization 


Share ot schools at target grade tunded by local or district level 


7 


Centralization 


Government level at which examinations implemented 


8 


Centralization 


Government level where curriculum determined 


9 


Special education 


Share ot schools at target grade classified as"special education schools’’ 


10 


Target grade 


Criteria tor promotion trom the target grade 


11 


Target grade 


Number ot subjects in target grade with standards; inclusion ot mathematics, 
science, or mother tongue in standards 


12 


Target grade 


System requirement tor mathematics at target grade 


13 


Target grade 


Mathematics emphasis on mastery, real lite, communication, and ICT 


14 


Target grade 


System requirement tor science at target grade 


15 


Target grade 


Science emphasis on mastery, real lite, communication, and ICT 


B. Teacher Preparation 


16 


Teacher preparation 


Degree requirements tor teacher certification 


17 


Teacher preparation 


ICT-specific requirements tor teacher certification 


18 


Teacher preparation 


Target grade teacher requirements tor in-service or protessional development 


19 


Teacher preparation 
topics 


Government subsidizes in-service or other protessional development on ICT 


C. Change in Past Five Years 


20 


Changes in last five years 


Five-year trends in ICT spending, pedagogyand assessment 


21 


Changes in last five years 


Five-year trends different in target-grade schools? 


22 


Changes in last five years 


Five-year trends different in mathematics and science? 


D. System-wide Poiicies and Practices on Use of iCT 


23 


ICT policies and practices 


National or system-wide policy? 


24 


ICT policies and practices 


Components (goals and policy types) ot national ICT policy 


25 


ICT policies and practices 


Approach to managing software and hardware 


26/27 


Language an obstacle 


Is language an obstacle in ICT use for teaching and learning? 


28 


ICT skills program 


Does system’s target grade have program for ICT-related skills? 


29 


ICT and pedagogy 


Does program include ICT with specific pedagogies? 


30 


21st-century skills policy 


Does system have program for 21st-century skills? 


31 


Websites on ICT in education 


List websites on ICT in education 


32 


Other relevant matters 


Are there other relevant features of the education system? 
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2.3 Administration, Coding, and Missing Data 

The NCQ_was administered online to the NRCs of the participating education systems. 
The data collection period was September to December, 2006. The final data file was 
taken on 20 December 2006, by which time all of the NRCs had completed many if not 
all of the items. Estonia and Slovenia did not answer a number of questions in the second 
half of the questionnaire, but otherwise the responses were adequately thorough. Both 
countries later clarified their answers to most of the questions to form a reliable basis for 
analysis and reporting. 

For each question the response alternatives were implicitly coded into numeric 
codes starting with “1” for the first category or checkbox, “2” for the second, and so 
forth. If responses to a question or question part were missing, then a code value of “9” 
was generally assigned. The exception was with two- or three-digit fields, in which case a 
“99” or a “999” was assigned. 

The lEAData Processing and Research Center (DPC) provided an initial data file 
in Excel format, coded as described above, and a final SPSS data file that included, after 
communication with the respective NRC and a review of the draft tables, augmented 
and supplemented data. Instances of non-response show as a “.” in the tables of Chapter 
3 of the international report. 

The full questionnaire with variable names can be obtained with the SITES 
2006 data files and user guide. A version of the questionnaire without variable names is 
available as an online appendix to the international report at http:/ /www.sites2006.net/ 
appendix. 

2.4 Indicator Development and Analysis 

Table 2.2 lists the indicators used in the analysis. The first eight indicators are named 
with the letter “u” and the remaining named with the letter “q”. The eight “u” indicators 
were taken from the Human Development Report 2006, obtained from the UNDP (2006), 
except for those countries for which there were no available data. The data were based on 
2004 information unless otherwise noted. 

The remaining indicators are either original responses to the NCQ_questionnaire 
or composites of these. The composites are all sum scores, except for one indicator, q4, 
which is based on an examination of the responses to Questions 4, 5, and 6. Responses to 
these questions determined if funding for education within each system came primarily 
from central sources, at the national or the provincial level, or from less centralized 
sources (i.e., lower, local levels). 

The NCQ_ contained a qualitative component, which meant that the statistical 
procedures used for the school and teacher components were not appropriate for this 
component. Because of the nature of the sample and the variables, the analysis consisted 
mostly of univariate and bivariate tabulations. In a few instances, sum scores were 
calculated by simply adding the responses to several items. 
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Table 2.2: Definitions for Indicators Used in the Analysis 



Indicator Name 


Indicator Label 


(Ul) 


Total population in millions 


(U2) 


Percentage of population in urban areas 


(U3) 


Gross domestic product (GDP) per person in US$ 


(U4) 


Inequality measured by subtracting the income of bottom io% from top io% in US$ 


(U5) 


Cellphone users are the number of users per 1,000 population in 2003 


(u6) 


Internet users are the number of users per 1,000 population in 2003 


(U7) 


"Education level" adds together the country literacy rate with the rate of gross enrolment from 
primary through tertiary systems 


(u8) 


“Educ.$” is the total public spending for education divided by the GDP 


(q4) 


Central funding combines responses from three NCQ questions (4, 5, & 6) by coding yes if the 
primary funding source was national or provincial; otherwise, it was coded no. Non-responses were 
coded 


(q?) 


Central control of curriculum components was based on question NCOy.This item was codedyes if 
the country had central or provincial control of three or four curricular components, but coded no if 
control was over one or two components only 


(qio) 


Criteria for promotion of students in target grade to next grade: a = national examination; b = school 
internal examination; c = oral and/or written examinations throughout the school year; d = portfolio 
of student work; e = other 


(qii) 


Number of subjects with attainment standards for target grade (NCQiii): none; all school subjects; 
only some subjects 


(qi6) 


Selection of teacher certification requirement options:i = postsecondary diploma and/or 
certification in education field; 2 = any postsecondary degree; 3 = any postsecondary degree plus 
certificate in education; 4 = other; 5 = requirements defined at local level only 


(qiy) 


ICT-specific requirements for certification: a = none; b = technical competence; c = subject teaching 
with ICT; d = ICT-based pedagogy; e = others; f= requirements defined at local level only 


(qi8) 


Sum of in-service or professional development components required of teachers (out of seven), 
based upon question NCQ181, items a-g 


(qig) 


Government subsidy of in-service or professional development for teachers in: a = ICT skills; b = use 
of ICT in subjects; c = use of ICT in administration; d = use of ICT for new approaches in learning 


(q2oa) 


The Index of Increased Spendingfor ICT is the sum of a series of seven questions that asked if each of 
seven types of ICT spending had decreased or increased in the past five years (NCQ201, items b-h) 


(q2oj) 


Increased New Pedagogical Practices is the sum ofthe series of six questions that asked if each of 
seven aspects of non-traditional practices had decreased or increased during the past five years 
(NCO201, itemsj-o). Item 201 was not included in this scale 


(q24) 


Number of ICT Policy Aspects is the sum of 11 questions (NCQ24) on each of 11 components (items 
a-k) of ICT policy 


(q25) 


Provision of Hardware/Software is directly derived from the choices to question NCQ25, which asked 
schools how they fund and acquire their hardware and software 


(q26) 


Language as an ICT Obstacle is directly derived from NCQ26 


(q28) 


ICT Skills at Target Grade is based on the answers to NCQ28 about the presence of a system-wide 
program on students' ICT-related skills 


(q29) 


The Number of New Pedagogies Using ICT sums responses to question NC02g (items b-f) 


(q3o) 


21st-century Skills Policy is the answer to question NCQ30, which focused on the presence or 
otherwise of system policy documents mentioning the promotion of 21st-century skills 



Note: See Tables 3. 1-3.4 of Chapter 3 ofthe SITES 2006 international report (Law, Pelgrum, & Plomp, 2008), pp. 44-55. 
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3. School-level 
Questionnaire Development 

Willem Johan Pelgrum 



3.1 Overview 

Two questionnaires were used at the school level. The first questionnaire, the “principal 
questionnaire,” was for school principals. The second, the “technical questionnaire,” 
was for persons, usually ICT coordinators, conversant with technology aspects at the 
school. 

This chapter describes the overall framework, the development process, and the 
final content of the school-level questionnaires. 

3.2 Concepts Addressed in the School-level Questionnaires 

Although, according to the conceptual framework presented in Chapter 1, the concepts 
at the school level were primarily intended for explanatory purposes, many of the school- 
level indicators had a descriptive value in themselves. For instance, the school-level data 
allow calculation of indicators concerning the availability of hardware (e.g., the student- 
computer ratio). These indicators have been, for manyyears, the core of ICT surveys and 
are of high political interest. However, as stated in Chapter 1, the school-level concepts 
in particular were intended to provide an explanatory context for the teacher-level 
concepts. Table 3.1 contains an overview of the concept- domains addressed in the two 
school-level questionnaires. 

Several sources constituted the basis for conceptualizing and operationalizing the 
school-level indicators. Among them were: 

• The teacher questionnaire: operationalizations of indicators on vision, support, and 
organization and management mirror comparable indicators used in the teacher 
questionnaire. (See Chapter 4 for the rationale and literature references that 
constituted the background of the teacher questionnaire.) 

• Previous assessments regarding ICT at the international level (Pelgrum & Anderson, 
2001) as well as at the national level (e.g., the Dutch ICT monitor: Sontag, van Haaf, 
van der Linden, & Meijs, 2004) : In particular, indicators regarding ICT infrastructure 
were copied from SITES-Ml and, where needed, updated in order to reflect the latest 
technology innovations (e.g., electronic whiteboards). The indicator on the presence 
oflifelong learning was also copied from SITES-Ml. 

• Previous research on ICT in education: BECTA (e.g., Jones, 2004; Scrimshaw, 2004) 
was a good source in terms of allowing summaries of earlier research findings (e.g., 
Bradley & Russell, 1997; Cuban, 1999; Lee, 1997; Pina & Harris, 1993) with regard 
to technical support, amount and quality of teacher training, willingness to change, 
and school leadership. 
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Table 3.1: Concepts Addressed in the School-level Questionnaires 



Concept(s) 


Description 


Infrastructure 


• Availability of ICT hardware (types of computers, local area network, internet connections, 
electronic whiteboards, etc.) 

• Availability of ICT software (general and subject-specific software, learning management 
systems, assessment tools, etc.) 

• Infrastructure needs 

• Problems 


Pedagogical practice 


• Extent to which lifelong learning practices are present in the school 


Vision 


• The vision of the school management with regard to pedagogy and ICT and covering three 
dimensions: traditional, lifelong learning (LLL), and connectedness 


Staff development 


• Encouragement extended to or requirements for teachers to acquire knowledge and skills 
with regard to pedagogical practices and the use of ICT 

• Priorities that school leadership gives to acquiring competencies 

• Ways that teachers in the school have acquired knowledge and skills for using ICT in 
teaching and learning 

• Availability (school-based and/or externally) of ICT-related courses 


Support 


• Persons involved in providing support and time-expenditure 

• Extent to which pedagogical support is available for teachers 

• Extent to which technical support is available for teachers 


Organization and 
management 


• Role of principals in initiating changes 

• Decisionmaking responsibilities 

• Management of change 

• Stimulation of cooperation between teachers 

• Promotion of alternative assessment practices 



• Theories about educational change: for example^ Fullan ( 1993). 

• The needs and expertise of the NRCs: during the concrete elaboration and 
operationalization of the instruments, NRCs offered valuable suggestions for wording 
of questions and answer options, based on their experiences in this domain of the 
study and their cultural background. 

The concepts that were operationalized in the school questionnaires constituted 

the basis for the construction of statistical indicators. These are described in more detail 

in Chapter 11. 

3.3 Questionnaire Development Process 

Construction of the school-level questionnaires involved several steps: 

i) The initial (draft) version was developed by the consortium led by the University of 
Twente on the basis of the project description included in the original tender. 

ii) The draft version was distributed to the NRCs to review in February 2005, well 
ahead of the first NRC meeting. This scheduling meant that the NRCs could 
study both the conceptual framework and the corresponding draft school-level 
questionnaires. During the meeting, participants contributed additional concepts, 
examples of operationalizations, and improvements to the proposed questions. 

iii) The draft version for the field trial took into account the suggestions made during the 
NRC review and was subsequently reviewed and commented on by the members of 
the consortium and the steering committee in April 2005. 
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iv) The resulting comments and suggestions were used to inform preparation of the 
second draft for the field trial, and this draft was also sent to the NRCs for review. 
The draft was accompanied with background documents describing in detail the 
actions taken in relation to the suggestions made at the first NRC meeting. 

v) After the field trial text had been copyedited (this work was organized by the lEA 
Secretariat) and laid out by the lEA DEC, the final version of it was sent to the NRCs 
in May 2005 for translation, followed by national and international translation 
verification, international layout verification, and field trial data collection. 

vi) The draft version for the main study was revised on the basis of field trial data 
available in October/November, 2005. The consortium then performed exploratory 
data analysis, including, but not limited to, the inspection of distributions, analysis 
of missing data, factor analyses, reliability analyses, and initial correlation analyses. 
The consortium identified and proposed, after consulting with the SITES steering 
committee, a list of main findings and changes per questionnaire item. The main 
study draft and the list were made available to the NRCs and intensively discussed 
during the second NRC meeting in December 2005. 

vii) The main study questionnaires were discussed and finalized by the consortium and 
steering committee, who took into account, wherever possible, the feedback and 
discussions from the NRC meeting. The questionnaires were sent to the NRCs in 
January 2006 after copyediting by the NRC of the Province of Alberta, Canada, and 
formatting and final layout by the lEA DPC. 

viii) As shown in the description above, the instrument development was a collaborative 
process in which the consortium took the lead by creating a first version of the 
questionnaires and in which the NRCs and the steering committee consecutively 
played an essential role during the further development and finalization on the basis 
of the conceptual framework. 

Table 3.2 summarizes the key dates and activities of this development. 

Table 3.2: Overview of the Instrument Development Process for the School-level 

Questionnaires 



Dates 


Activities 


September to November 2004 


• Initial item development 


December 2004 


• First steering committee meeting 


February to April 2005 


• First NRC meeting (Enschede, the Netherlands) 

• Second steering committee meeting 

• Revision of school-level questionnaires based on input received from NRCs 


May to August 2005 


• Finalization of questionnaires for the field trial 

• Field trial instruments to NRCs for translation and subsequent operations 


August to October 2005 


• Field tria 1 (17 cou ntries) 


November to December 2005 


• Analysis of field trial results and draft recommendations on revision of 
questionnaires for NRC meeting 


December 2005 


• Second NRC meeting (Phuket, Thailand) 

• Third steering committee meeting 

• Review of field trial findings and finalization of recommendations on main study 
instruments 


January 2006 


• Finalized MS questionnaires to NRCs 
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3.4 Contents of the Main Study School-level Questionnaires 

The final version of the questionnaire for school principals contained 34 questions, 
covering 222 variables. The final version of the technical questionnaire contained 19 
questions, addressing 115 variables. On the basis of the field trial results and feedback, 
the consortium estimated that the questionnaires would take approximately 30 to 45 
minutes to complete. 

Table 3.3 contains a description of the content of the questionnaire for school 
principals, while Table 3.4 contains a description of the content of the questionnaire for 
ICT coordinators. 



Table 3.3: Content of the Main Study Questionnaire for School Principals 



Question Number 


Question Content 


Description 


1 


Presence of pedagogical practices 


Twelve statements about pedagogical practices 
Note:this question overlaps with SITES-Mi 


2 


Pedagogical vision 


Ten statements related to pedagogical goals 


3 


ICT vision 


Ten statements about the importance of ICT use 


4 


Organization: cooperation 


Five statements about the school leadership's 
encouragement of ICT use 


5 


Organization: school policy 


Twelve statements about policy actions taken by the 
school 


6 


Infrastructure: priorities 


Eleven statements about priorities for infrastructure 




Staff development: priorities 


Staff development, and organizational measures to 
stimulate ICT use 


7 


Organization: change management 


Eleven statements related to several aspects of change 
management (re-allocating workloads, internal 
communication) 


8 


Organization: change management 


Eleven statements about the extent to which the school 
leadership promoted active ways of student learning 


9 


Organization: change management 


Nine statements about the extent to which the school 
leadership undertook action to promote internal 
communication 


10 


Organization: change management 


Four statements about the extent to which the school 
leadership encouraged cooperation among teachers 
(inside and outside the school) 


11 


Organization: change management 


Eight statements about the extent to which the school 
leadership encouraged the use of traditional and 
alternative forms of assessment 


12 


Staff development: requirements 


Ten statements about competency areas in which 
teachers are encouraged or required to acquire 
knowledge and/or skills 


13 


Staff development: needs of school 
leaders 


Ten statements about priority areas in which the 
school leadership needs to acquire competencies 


14 


Support: pedagogical 


Types of persons who provide (frequency) pedagogical 
support 


15 


Support: pedagogical 


Statements about extent to which pedagogical support 
is available for six different teaching/learning activities 


16 


Infrastructure: obstacles 
Staff development: obstacles 
Other obstacles 


Extent to which school leaders perceive 15 obstacles as 
seriously hindering the school's capacity to realize 
its pedagogical goals 
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Table 3.3: Content of the Main Study Questionnaire for School Principals (contd.) 



Question Number 


Question Content 


Description 


17 


Organization: grouping 


Four statements about extent to which students are 
grouped in the school 


18 


Organization: flexibility 


Five statements about extent to which student groups 
are fixed and/or flexible 


19 


School characteristics: size 


Number of boys and girls enrolled in the school 


20 


School characteristics: grade coverage 


The highest and lowest grades in the school 


21 


School characteristics: urbanization 


Area in which the school is located 


22 


School characteristics: absenteeism 


Percentage of students a bsent on a ty pica 1 school day 


23 


School characteristics: student body 


Percentage of native speakers at school 


24 


School characteristics: experience 
of innovation 


Recent involvement of school in six types of innovation 


25 


School characteristics: locus of control 


Inquiry about the primary responsible agency/person 
(four options) for six decisionmaking areas 


26 


Principal background: innovation 
involvement 


Eight statements about the involvement of the school 
principal in initiating change 


27 


Principal:experience 


Number of years of experience of the school principal 


28 


Principal: age 


Age of the principal (five intervals) 


29 


Principal:gender 


Gender of the principal 


30 


Principal:funding involvement 


Four statements about the principal's involvement in 
raising ICT-funds for school 


31 


Principal: use of ICT 


Frequency with which the principal personally used a 
computer 


32 


Principal: use of ICT 


Ten statements about the type of activities for which the 
principal used a computer 


33 


Principal: home access 


Question about access to computer at home 


34 


Principal: home use 


Two statements about how principal used the computer 
at home 



Table 3.4: Content of the Main Study Questionnaire for ICT Coordinators 



Question Number 


Question Content 


Description 


1 


Use of ICT: number of years 


Number of years school has been using ICT forteaching 
and learning 


2 


Use of ICT: stage 


Where school fits within five stages of ICT introduction 


3 


Use of ICT: subjects 


Frequency of use by students of ICT in six school subjects 


4 


Infrastructure: software 
Infrastructure: hardware types 


Inquiry about need for and availability of seven software 
types, four hardware types, and mail accounts for 
teachers and/or students 


5 


Infrastructure: quantity 


Open questions about number (for students in grade 
range and total, for teachers only, administration only) 
and type of computers (internet-connected, LAN- 
connected, multimedia) available in the school 


6 


Infrastructure: quantity 


Number of laptops in the school 
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Table 3.4: Content of the Main Study Questionnaire for I CT Coordinators (contd.) 



Question Number 


Question Content 


Description 


7 


Infrastructure: sundries 


Number of PDAs, graphic calculators, smart-boards, and 
beamers available in the school 


8 


Infrastructure: student-owned 


Percentage of students bringing their own PDAs, graphic 
calculators, and laptops to school 


9 


Infrastructure: location 


Location of computers 


10 


Organization: maintenance 


Types of persons involved in maintenance 


11 


Staff development: availability 


Types of channels (list of 10) through which staff 
development takes place 


12 


Staff development: providers 


Internal and/or external availability of seven types of 
courses 


13 


Characteristics of respondent 


Which of six positions the respondent holds 


14 


Characteristics of respondent 


Duties (list of six) of respondent 


15 


Support: availability 


Number of hours per week by eight different types of 
person 


16 


Support: availability 


Eleven statements about teaching/learning activities for 
which a certain extent of technical support is available 


17 


Obstacles 


See principal questionnaire item 16 


18 


Coordinator: home access 


Question about respondent's access to computer at home 


19 


Coordinator: home use 


Two statements about how technology coordinator uses 
computer 
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4- Teacher Questionnaire 
Development 

Nancy Law 
Angela Chow 



4.1 Overview 

A key focus of the SITES 2006 survey has heen to gain an overall understanding of 
teaching and learning practices at the lower secondary level in schools around the world, 
howICTisusedinthem,andwhatfactorsare associated with variouspedagogicalpractices 
and ICT use. The SITES research consortium assumed that pedagogical practices and 
ICT use in a specific curriculum context (i.e., for a given subject at a particular level in 
a specific classroom) are affected by teachers’ characteristics, school-level factors, and 
system-level factors, as well as by the characteristics of the students involved. It is also 
assumed that the pedagogical practices and the students’ characteristics interact to give 
rise to different learning outcomes. Teachers’ pedagogical practices and the use of ICT 
in them therefore lie at the heart of the study. 

The core component of the teacher questionnaire was designed to collect 
information about these practices as well as to identify the system factors associated with 
different approaches and ICT use. The last part of the teacher questionnaire (Section 
VIII) was designed to examine in-depth descriptions from teachers of their satisfying 
experiences when using ICT in “micro-pedagogical contexts.” This exploratory part of 
the study was conducted as an international option; its design and findings are reported 
in full in Chapter 7 of the international report (Voogt, 2008). 

This chapter focuses on the design and development of the core component of 
the teacher questionnaire, that is. Sections I to VII. 

4.2 Concepts Addressed in the Teacher Questionnaire 

An important purpose of SITES has been to investigate the characteristics of teachers’ 
pedagogical practices and ICT use, as well as how these are related. The SITES-M2 
findings indicated that curriculum goals, the roles played by teachers (as reflected by 
their practices), and the roles played by students relative to their learning practices are 
the three aspects most indicative of the pedagogical approach of the teacher. 

Three sets of core indicators on pedagogical orientation were therefore developed, 
namely the curriculum goal orientation, the teacher practice orientation, and the student 
practice orientation. These indicators were constructed on the basis of the teachers’ 
responses to questions on the relative importance of a range of curriculum goals and the 
relative frequency of occurrence for a range of teacher and student activities respectively. 
In addition, for each item on the list of teacher and student activities, teachers were asked 
to indicate whether ICT had been used in those activities. This latter set of responses 
was used to compute two further sets of core indicators for the pedagogical orientation 
of (respectively) teacher and student practices involving ICT use. An additional core 
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indicator was whether the teacher had actually used ICT at all with the target class within 
the school year during which the survey was conducted. 

To provide a comprehensive picture of classroom situations, in addition to the 
core indicators, further survey questions were designed in the teacher questionnaire to 
provide indicators on 

• The methods of organizing teaching and learning 

• The location and time when teaching and learning occurred 

• The learning resources (including traditional and digital) used 

• Assessment practices 

• Whether ICT had heen used with the target class. 

Those teachers who had used ICT with the target class were asked to assess the 
perceived impact of pedagogical ICT use on themselves and on their students. Teachers 
were also asked about the degree of priority the school accorded use of ICT in their 
teaching in the following academic year. These indicators are referred to as supplementary 
indicators. 

Several personal and contextual factors can influence teachers’ pedagogical ICT 
use. Personal factors include: 

• Demographic background (age, gender, academic and professional qualification, and 
professional experience) 

• Technical competence and competence in using ICT for pedagogical purposes 

• Teaching philosophy/ pedagogical beliefs and rationale for using ICT. 

Contextual factors include: 

• Teachers’ reported availability and participation in ICT-related professional 
development activities 

• Teachers’ perceptions of obstacles to using ICT in their teaching 

• Teachers’ perceived presence of a community of practice in their respective schools. 

Indicators derived from the questions relating to these personal and contextual 
factors provide explanatory indicators for the study because they aid the development of 
an explanatory model for teachers’ pedagogical ICT use. 

One important assumption underpinning the design of the study is that teachers’ 
decisions on which forms of ICT to use in their teaching and how to use these depend 
not only on the school subject taught but also on the characteristics of the students 
they teach. Hence, the research consortium considered it very important in terms of 
questionnaire design that when the teachers answered the questions related to the core 
and supplementary indicators, they were quite clear that their answers had to refer to 
a specific class that they were teaching in the school year during which the survey was 
conducted. A target class accordingly was identified for each of the teachers sampled in 
the study (also see Section 4.3), and the teacher questionnaire was developed so that 
it began with questions about the target class. These questions included demographic 
information (number of students in the class, gender mix, academic track, percentage 
of absenteeism, and percentage of students who are native speakers of the language of 
instruction), students’ ICT skills, and students’ access to ICT outside of the school. 

Table 4. 1 summarizes the core supplementary and explanatory indicators that 
constitute the framework for the teacher questionnaire. The design of the teacher 
questionnaire was also informed by an extensive literature review of various national and 
international studies on teachers’ practices using ICT. Table 4.1 therefore also provides 
the key references that informed the development of the questionnaire. 
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Table 4.1: Summary of Indicators, Concepts, Key Literature, and Question Numbers in the 
Core Component of the Main Study Teacher Questionnaire 



Type of 
Indicators 


Concepts 


Key Literature 


Question 

Number 


Core 


Curriculum goal orientation 


Becker and Anderson (1998); 
Kozma and McGhee (2003); 
Law et al. (1999, 2003) 


8 




Teacher practice orientation 


Henke et al. (1999); 

Kozma and McGhee (2003); 
Law et al. (2003); 

Mullens and Gayler (1999) 


14 




Student practice orientation 


Kozma and McGhee (2003); 
Law et al. (1999, 2003) 


16 


Supplementary 


Organization of 
teaching and 
learning activities 


Teaching methods 


Kozma and McGhee (2003); 
Law et al. (2003) 


9 






Separation of teachers 
and learners in spatial 
locations and time for 
teaching and learning 
activities 


Law et al. (2003) 


10, 11, 12, & 13 




Learning resources and ICT used 


Kozma and McGhee (2003); 
Law et al. (1999, 2003) 


17 




Assessment practices 


Henke et al. (1999); 

Kozma and McGhee (2003); 
Law et al. (2000, 2003) 


15 




Impact of ICT use on teachers and students 


Becker and Anderson (1998); 
Law et al. (2003) 


19, 20 


Explanatory 


Teacher characteristics 


Technological and 
pedagogical competence 


Becker and Anderson (1998); 
Law et al. (1999) 


21 






Teacher's pedagogical 
vision of ICT use in near 
future 


Law et al. (1999) 


22 




Teacher perceptions of 
contextual factors 


Obstacles to ICT use as 
perceived by teachers 


Becker and Anderson (1998); 
Law et al. (1999) 


23 






Participation and 
intention to attend 
professional 
development activities 


Becker and Anderson (1998); 
Law et al. (1999) 


24 






Presence of various 
aspects of community of 
practice in school 


Dexter and Anderson (2002); 
Dexter, Seashore, and Anderson, 
(2002); 

Geijsel et al. (2001) 


25,26, 27, & 28 




Student characteristics 


Demographic 
information about the 
target class:class size, 
gender mix, curriculum 
track, absenteeism, 
mother tongue in 
relation to medium of 
instruction, contact 
hours per week, and 
students' ICT skills 


n/a 


1-7 
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The remainder of this section gives a detailed description of the key aspects of the 
design for the main components of the teacher questionnaire. 

4.2.1 Pedagogical Practice and ICT Use 

SITES 2006 built on the design considerations and findings of the previous two SITES 
modules in developing a framework for categorizing and comparing pedagogical 
practices. The terms “emerging paradigm” and “innovative practices” embody a strong 
focus on change. One important aspect of change in this context concerns the policy 
goals that underpin the system-level support given to ICT use in schools. While these 
goals may differ across countries because of different priorities given to educational goals, 
one common goal that has emerged in recent years in many policy documents/education 
master plans is the use of ICT to support national education reform efforts aimed at 
preparing students for the challenges of the 21st century (Pelgrum & Law, 2003). 

As the world moves toward a global and knowledge-based economy, many 
societies are experiencing changes in the ability profile of their human resources (Riel, 
1998). The reform goals of these societies now focus less on learners’ specific knowledge 
and/or skills and more on their metacognitive and affective qualities (see, for example, 
Danish Ministry of Education, 1997; Education and Manpower Bureau HKSAR, 1998; 
Singapore Ministry of Education, 1997). Metacognitive qualities include creative 
thinking, lifelong learning abilities, and ability to cooperate and communicate. Affective 
qualities include a sense of social responsibility, which encompasses value judgments 
and behavioral norms in cyberspace, and readiness to understand other cultures and 
lifestyles. 

4.2.2 Emergent Practices and Pedagogical Approaches 

When considering the existing literature on ICT in education and how ICT use in an 
industrialized society differs from that in an information society, we have at hand several 
ways of describing the desired features of classrooms in the 21st century. 

SITES-Ml and SITES-M2 used the term emerging pedagogy (as opposed 
to traditionally important pedagogy) to describe the consensus on desirable learner 
characteristics and learning contexts that seem to be emerging. These include students’ 
increasing autonomy in determining learning goals and learning strategies, and the 
presence of more open-ended authentic problem contexts for enquiry-based learning 
and learning in collaborative teams. Both the SITES-Ml and SITES-M2 results (Kozma, 
2003; Law, Chow, & Yuen, 2005; Law, Yuen, Ki, Li, & Lee, 2000; Mioduser, Nachmias, 
Tubin, & Forkosh-Baruch, 2003; Pelgrum & Anderson, 1999) indicated that significant 
changes had taken place in the roles played by teachers and learners in some of the 
innovative practices. Classrooms that had leveraged the potentials of ICT to foster the 
development of 21st-century skills in students tended to be student-centered rather than 
teacher-centered (as seen in traditional classrooms). These classrooms also tended to 
be relatively well connected to people outside of the classroom/school walls (thus less 
isolated than previously), and the learning process tended to contribute more often to 
productive outcomes, such as solving authentic problems. The SITES-Ml and SITES- 
M2 surveys were designed and conducted against the backdrop of a general desire to 
realize the above broad reform goals. 

Based on the above review, the SITES consortium considered it appropriate to 
move on from the dichotomy used in SITES-Ml and to use more specific categorizations 
ofpedagogical practices in the SITES 2006survey. In the finalSITES2006instrumentation, 
pedagogical practice characteristics were categorized into three pedagogical practice 
orientations: traditionally important, lifelong learning, and connectedness. 
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Traditionally important takes the same definition as that used in the SITES- 
M1 study. Lifelong learning and connectedness are both associated with those practice 
characteristics that aim to develop students’ abilities to meet the challenges of life in 
the 21st century as described in many of the education policy documents calling for 
system-wide reforms in curriculum and pedagogy that have appeared since the mid 
1990s. However, these two terms differ in their emphases. Lifelong learning is concerned 
more with the development of students’ interests and capacity to make collaborative 
inquiries on authentic problems; the connectedness orientation concerns linking students 
and teachers with peers and experts outside of the school walls. 

4.2.3 Key Aspects for Understanding and Describing Pedagogicai Practices 

When designing the questions in the teacher questionnaire, the development team 
had to determine which of these would produce data indicating 21st-century-oriented 
teaching and which would produce data signifying traditional teaching. In analyzing the 
S1TES-M2 international cases. Law et al. (2003, 2005) identified six dimensions along 
which the ICT-using innovative practices could be compared in terms of their extents 
of innovativeness. Of these, three were considered most indicative of the pedagogical 
paradigm of a teaching practice: 

• Curriculum goals 

• Teacher practices (teachers’ roles) 

• Student practices (students’ roles). 

These three dimensions were thus selected as core indicators for SITES 2006. 
In addition, questions were designed to gather data for the more factual, “observable” 
features of pedagogical practices, that is, the methods used to organize learning and 
teaching, and the learning resources and assessment practices ICT used. These, the 
team determined, could be analyzed to provide supplementary indicators for teachers’ 
pedagogical practices. 

4.2.4 iCT Use in Pedagogicai Practices 

One research area of particular interest within SITES 2006 is what impact, if any, 
integration of ICT in the teaching and learning process has on the pedagogical orientation 
and other aspects of teachers’ practice. To explore this matter, two sets of responses were 
solicited from SITES respondents via four questions related to teaching and learning 
practices: 

• Teacher practices, planning and regulation, execution (teachers’ roles) 

• Student practices (students’ roles) 

• Methods of organizing teaching and learning 

• Assessment practices (types of assessment). 

One set of responses. Part A, concerns the frequency with which certain activities 
take place. The other set of responses. Part B, determines if ICT is used in such activities. 
Figure 4. 1 provides an excerpt from the items on teacher practice in order to illustrate the 
design of the related question. 
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Figure 4.1 : Excerpt from Question 14 of the Teacher Questionnaire to Illustrate the Two 
Parts to the Question 



14. 


In your teaching of the target class in 


this school year: 












(a) How often do you conduct the following? 

(b) Do you use ICT for these activities? 

Please mark only one choice for each of the two parts in 


each row. 














(a) How often do you conduct the following? 


(b) ICT used? 






1 


2 


3 


4 


1 


2 






Never 


Sometimes 


Often 


Nearly 


No 


Yes 












always 






A 


Present information/demonstrations 
and/ or give class instructions . . . 


□ 


□ 


□ 


□ 


□ 


□ 


B 


Provide remedial or enrichment 
instruction to individual students 
and/or small groups of students ... 


□ 


□ 


□ 


□ 


□ 


□ 


C 


Help/ advise students in exploratory 
and inquiry activities . . . 


□ 


□ 


□ 


□ 


□ 


□ 



Chapter 1 1 describes the development of the scales for pedagogical approach and 
for comparing the impact of ICT on pedagogical approach. 

4.3 Target Class and Sequencing of the Questionnaire Components 

To ensure that respondents would be made aware of the meaning of the target class and 
when they needed to answer with reference to the target class, several design features 
were implemented. 

• The cover page of the teacher questionnaire stated a specific, randomly selected Grade 
8 target class that the teacher needed to think about when answering the question. 
Identification of the target class also provided the subject context (mathematics or 
science) that the teacher had to think about. 

• The beginning of the teacher questionnaire provided a passage for respondents that 
read as follows: “When a question refers to the ‘target class,’ please think only about 
the class/ course you are teaching in this school year that is specified on the cover page. 
You will answer all questions with reference to the teaching of the subject (domain) 
that is specified on the cover page in this class.” 

• The first section of the teacher questionnaire (i.e.. Section I) asked respondents 
about student characteristics for the target class, followed by sets of questions on 
goals, practices, resources, and perceived impact of ICT use. 

• Questions about teachers’ personal characteristics and theirperceptions of contextual 
factors were placed in the last section of the core part of the teacher questionnaire 
(i.e.. Section VII). The reason for this placement was that these questions were not 
specific to the target class. 

4.4 Questionnaire Development Process 

The development of the SITES 2006 teacher questionnaire took place from September 
2004 to January 2006, parallel to the development of the school-level questionnaires 
(see Chapter 3). Because the SITES surveys had not previously included a teacher 
component, portions of the 2006 questionnaire had to be newly developed. To support 
and inform this process prior to the field trial, a pilot test of the teacher questionnaire 
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was carried out in Hong Kong in late 2004, and selected items were administered to 
a convenience sample of teachers in 12 countries as part of an online data collection 
(ODC) try-out in April 2005 (see Section 8.5). 

The developmental work was undertaken hy a research team from the Centre 
for Information Technology in Education, University of Hong Kong (CITE), led hy 
Nancy Law, with input from the international study consortium members, the steering 
committee members, and the NRCs. 

Starting from September 2004, intensive development work took place at CITE 
to develop the first draft instrument. The main design challenge at the beginning stage 
of instrument development involved exploring and determining if questions that would 
yield reliable scale scores on pedagogical orientations as described in Section 4.2 above 
could be developed for all participating systems. 

Given the time constraints on the study, it was not possible to conduct an 
international pilot testing. To ensure that there would be a reasonably good set of 
questionnaire items for international field trialing, about 85 teachers in Hong Kong were 
invited to complete the pilot instrument, which included 23 questions in eight sections, 
with respect to a specific class they were teaching at the time. The data collected from 
the pilot test was analyzed to check if the reliabilities for the intended scale items met 
the necessary statistical requirements. The preliminary result was positive, and the data 
were used by the CITE team to refine the instruments. Further details about the scale 
development work at this stage are reported in Chapter 11. 

The proposed field trial teacher questionnaire was discussed and modified 
alongside the school-level questionnaire during the first NRC meeting in order to arrive 
at a consistent and integrated set of instruments. After copy-editing and final layout for 
both the print and online versions, the final field trial questionnaires were sent to NRCs 
in May 2005 for translation, translation verification, and layout verification. The NRCs 
also received at this time a summary of all changes made to the draft instrument as a 
result of the discussions at the first NRC meeting. 

The field trial was conducted from August to October 2005, and the resulting data 
were used for the following four purposes: 

• To check whether the three core indicators (curriculum goals, teacher practice, and 
student practice) yielded statistically reliable scales; 

• To identify which core, supplementary, and explanatory indicators demonstrated 
good psychometric properties and a high likelihood of contributing to a good model 
of pedagogical practice and ICT use; 

• To consolidate and refine the questionnaire in terms of linguistic clarity and length; 
and 

• To provide a concrete set of data for exploring the kinds of analyses to be conducted 
and reported in the main study. 

The 43 questions that addressed the 314 variables in the field trial teacher 
questionnaire covered eight sections: (l) target class; (ll) curriculum goals; (ill) teacher 
practices; (IV) student practices; (V) learning resources and technology infrastructure; 
(Vl) impact of ICT use; (VIl) demographic information about teachers and the schools 
in which they worked; and (VIIl) one specific teaching experience that used ICT 
(international option). 

The sections on target class information and on one specific ICT-using teaching 
experience were added after the first steering committee and NRC meetings. Nearly 
1,900 teachers from schools in 17 education systems participated in the field trial. The 
descriptive statistics were checked in order to identify problematic items. Analysis 
was also conducted to identify the optimal selection for constructing scales with the 
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minimum number of items. The field trial analysis results were presented and discussed 
during the second NRC meeting held in December 2005, in Phuket, Thailand. 

To ensure a high completion rate, the instrument development team agreed that 
none of the main study instruments must take longer than 40 minutes for the respondents 
to complete. The field trial instrument was slightly longer than the anticipated length of 
the final teacher questionnaire so as to provide statistical information for the selection 
of the final items. Based on the field trial results, the team not only reduced the number 
of questions from 43 to 41 but also substantially reduced tbe number of items in some 
of the questions in order to shorten the questionnaire. The team also decided that the 
section on one specific ICT-using teaching experience would become an international 
option to be placed at the end of the questionnaire. 

The final version of the main study teacher questionnaire was made available to 
NRCs in January 2006. Table 4.2 summarizes the dates and activities associated with its 
development. 



Table 4.2: Overview of the Questionnaire Development Process for the Teacher Questionnaire 



Dates 


Activities 


September to November 2004 


• Initial item development 

• Pilot test (Hong Kong) 


December 2004 


• First steering committee meeting 


February to April 2005 


• First national research coordinators' (NRCs) meeting (Enschede, the Netherlands): 
review of study framework and instruments 

• Second steering committee meeting 

• Revision of teacher questionnaire based on input received from NRCs 


April 2005 


• Try-out of selected items for ODC (online data collection), 12 countries 


May to August 2005 


• Finalization of questionnaire for the field trial 




• Field trial instruments to NRCs for translation and subsequent operations 


August to October 2005 


• Field trial (17 countries) 


November to December 2005 


• Analysis of field trial results and draft recommendations on revision of 
questionnaire for NRC meeting 


December 2005 


• Second NRC meeting (Phuket, Thailand) 

• Third steering committee meeting 

• Review of field trial findings and finalization of recommendations on main study 
instruments 


January 2006 


• Finalized MS questionnaire to NRCs 



4.5 Contents of the Main Study Teacher Questionnaire 

The final teacher questionnaire consisted of eight sections containing 41 questions and 
collecting data on 271 variables. The development team estimated that respondents 
would need approximately 30 to 40 minutes to complete the questionnaire. Table 4.3 
presents a summary of all questions in the final instrument. The questionnaire wording 
was generic enough to suit both teachers of mathematics and teachers of science, 
although the actual subject context for teachers was clearly identified on the front page 
of the questionnaire. 
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Table 4.3: Questions in the Core Component of the Main Study Teacher Questionnaire 



Question 

Number 


Section 


Question Content 


Description 


1 


1 


Class size 


Number of students in the target class 


2 


1 


Gender mix of the class 


Whether the target class is attended by boys only, girls only, or both 
genders 


3 


1 


Curriculum track 


Whether the target class is in academic track, vocational track, or 
with no tracking 


4 


1 


Absenteeism 


Approximate percentage of students absent from target class on a 
typical school day 


5 


1 


Language of instruction 


Approximate percentage of students in the target class whose native 
language is the language of instruction 


6 


1 


Class contact hours 


Hours of scheduled class time per week teacher spends on teaching 
the target class mathematics/science 


7 


1 


Students' ICT skills 


Proportion of students in the target class who have competence 
in various ICT skills: word-processing, database software, spreadsheet, 
presentation software, application of multimedia, email, internet, 
graphic calculator, and data-logging tools 


8 


II 


Curriculum goals 


Teacher's perception about the importance of various curricular 
goals. This question yielded three core indicators on curriculum goal 
orientation:"traditionally important,”"lifelong learning,” and 
“connectedness” 


ga 


III 


Teaching methods 


Frequency with which teacher carries out various teaching and 
learning activities during the target class's scheduled learning time 


gb 


III 


ICT use in teaching 
method 


Whether ICT is used when various teaching and learning activities 
take place 


lO 


III 


Separation of teacher/ 
learners 


How often the teacher and students are separated in spatial 
location when instruction takes place 


11 


III 


Separation of teacher/ 
learners 


How often students are separated relative to one another in 
spatial location when participating in planned learning activities 


12 


III 


Separation of teacher/ 
learners 


How often students' learning activities take place outside of 
scheduled school hours 


13 


III 


Separation of teacher/ 
learners 


How often the teacher provides feedback to students in the target 
class outside of scheduled school hours 


14a 


III 


Teacher practices 


Frequency with which various teaching activities take place. This 
question yielded the three core indicators for teacher practice 
orientation 


14b 


III 


ICT use in teacher 
practices 


Whether ICT is used when the various teaching activities take place. 
Responses to this component of the question yielded the core 
indicators for ICT-using teacher practice orientation 


15a 


III 


Assessment practices 


Whether the teacher uses various assessment methods when 
assessing students' performance 


15b 


III 


ICT use in assessment 


Whether the teacher uses ICT when carrying out the different kinds 
of assessment 


16a 


IV 


Student practices 


Frequency with which students participate in various activities. 
This question yielded the three core indicators for student practice 
orientation 


16b 


IV 


ICT use in student 
practice 


Whether students use ICT when participating in various activities. 
Responses to this component of the question yielded the core 
indicators for ICT-using student practice orientation 
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Table 4.3: Questions in the Core Component of the Main Study Teacher Questionnaire (contd.) 



Question 

Number 


Section 


Question Content 


Description 


17 


V 


Learning and ICT 
resources used 


How often the teacher incorporates various learning resources and 
tools when teaching his/her target class 


18 


VI 


Whether ICT has been 
used with the target 
class 


Whether ICT is used at all with the target class. This question was a 
core indicator as well as a screening question for questions applicable 
onlyto ICT-using teachers 


19 


VI 


Perceived impact of ICT 
on self 


The teacher's perception of the extent of various impacts of ICT use 
on himself/herself 


20 


VI 


Perceived impact of ICT 
on students 


Teacher's perception of extent of various impacts of ICT use on 
students 


21 


VII 


Teacher self-reported 
ICTskills 


Teacher's self-reported levels of confidence in accomplishing various 
general and pedagogical uses of ICT 


22 


VII 


Teacher vision of ICT use 
in the near future 


Teacher's priorities for different pedagogical uses of ICT within the 
next two school years 


23 


VII 


Obstacles to ICT use 


Whether teacher had experienced various obstacles to ICT use when 
teaching 


24 


VII 


Course participation in 
ICT-related professional 
development 


Whether teacher had participated in or would like to attend various 
types of ICT-related professional development activities 


25 


VII 


Presence of community 
of practice in school 


Teacher's perceptions of extent to which school staff hold a shared 
vision 


26 


VII 


Presence of community 
of practice in school 


Teacher's perception of extent to which school staff experience 
shared participation in decisionmaking 


27 


VII 


Presence of community 
of practice in school 


Teacher's perceptions of extent to which teachers engage in 
professional collaboration in the school 


28 


VII 


Presence of community 
of practice in school 


Teacher's perceptions of extent to which various kinds of support are 
available in the school 


29 


VII 


Computer access at home 


Whether teacher has access to a computer at home 


30 


VII 


Uses of computer at home 


Whether teacher uses his/her home computer for teaching-related 
activities and/or connectingtothe internet 


31 


VII 


Teacher's age 


The age group the teacher belongs to 


32 


VII 


Teacher's gender 


Teacher's gender 


33 


VII 


Teacher's academic 
qualification 


Highest level of education attained by the teacher 


34 


VII 


Teacher's academic 
qualification 


Whether the teacher has a Bachelor's degree in mathematics and/or 
science 


35 


VII 


Teacher's professional 
qualification 


Whether the teacher has a teaching license or a certificate 


36 


VII 


Teacher's teaching 
experience 


Number of years of experience teacher has in teaching mathematics 
or science (select appropriate range) 
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5. Translation, National Adaptation, 
and Verification 
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5.1 Overview 

The international version of the SITES 2006 questionnaires was developed and prepared 
in English by the SITES 2006 consortium with contributions from the NRCs. The 
questionnaires were subsequently translated by the participating countries into their 
languages, 18 in total (see Table 5.1). The translation process was designed to ensure 
the best possible translation quality and appropriate adaptation for the national context 
while ensuring international comparability. 

Each country was expected to follow the procedures for the translation of 
questionnaires into the national language(s) and cultural context(s). The guidelines were 
provided to all NRCs in the SITES 2006 Survey Operations Manual (lEA DEC, 2006), 
prepared by the lEA Data Processing and Research Center (DPC) in collaboration with 
the SITES consortium and elaborated at the NRC meetings. 

The translated and/or adapted questionnaires were checked in a rigorous 
process of translation verification and adaptations review. This process was intended to 
confirm, through direct comparisons, that the translated materials were equivalent to 
the international versions, and it was managed by the lEA Secretariat in Amsterdam, the 
Netherlands, which cooperated with Lionbridge Inc. (London, Dublin, and Brussels) to 
verify the translations for each of the education systems. Verified instruments were then 
returned to the NRCs with suggestions for changes and improvements. In addition, the 
international coordinating center (ICC) reviewed and noted any discrepancies between 
the layout of the translated instruments and the international version, and gave final 
approval for printing and administering the materials. 

Each participating education system was asked to submit materials for verification 
before the field trial and before the main data collection. This two-stage process allowed 
each system to be assured of a set of translated materials that had been reviewed on two 
separate occasions and had the assistance of diagnostic statistics, which were used to 
identify and remedy mistranslated questions before the main data collection. 

All SITES 2006 participants complied well with the requirements for translation 
and national adaptation verification of the survey instruments. Language verification 
was conducted twice for a majority of the participants: (i) before the field trial, and (ii) 
before the main data collection. In the case of the two Canadian provinces (Alberta and 
Ontario) and South Africa, which joined SITES 2006 late, the verification was conducted 
only before the main data collection. 
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Table S.l : Languages Used for the SITES 2006 Instruments 




5.2 Translating and Adapting the SITES 2006 Instruments 

5.2.7 Survey Languages 

The majority of the SITES 2006 participating systems used only one language for 
administering the survey. Three participants — Estonia, Finland, and Ontario (Canada) — 
used two languages each. The translation/ adaptation process for these countries required 
careful checking to ensure that both versions of the national instruments were equivalent. 
The NRCs were advised to involve, during preparation of the national instruments, a 
special reviewer — a person familiar with both languages — to check the comparability 
of the two versions. 

5.2.2 Instruments to be Translated 

For SITES 2006, the materials requiring translation included: 

• The teacher questionnaire 

• The principal questionnaire 

• The technical questionnaire. 
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To meet the needs of an education system electing to collect data online and in 
paper mode, the cover letters (equivalent to the first two pages of the questionnaire) 
prepared for teachers, school principals, and ICT coordinators also required translation. 

Education systems administrating the questionnaires in English did not have to 
translate the instruments hut were required to adapt the international English of the 
original versions to the variant of English appropriate for their country, including any 
necessary cultural adaptations. 

5.2.5 Translator and Reviewer 

Each SITES 2006 participant was advised to appoint a team of two persons to work on 
the translation and adaptation of the instruments: a translator and a reviewer. 

Translators were expected to have an excellent knowledge of hoth English and the 
target language, and — if possible — experience in the educational context of the survey 
as well as some familiarity with survey development in general. Reviewers (who were 
also expected to have an excellent knowledge of hoth English and the target language) 
were required to he well experienced with the country’s educational context. 

After the instruments had heen translated and adapted, the reviewer checked the 
quality of the translation and if the translation was appropriate for the target populations 
and context. The NRC then analyzed the reviewer’s suggestions and incorporated them 
into the translation, as he or she deemed necessary. In situations in which more than one 
translator and one reviewer worked on the national version of the instruments, additional 
checking was required to ensure consistency of the translation/adaptation within and 
across the instruments. 

5.2.4 Translation and Adaptation Guidelines 

To ensure that appropriate translations and adaptations were made during production of 
the SITES 2006 questionnaires, the ICC provided basic guidelines for these processes in 
the SITES 2006 Survey Operations Manual (lEA DEC, 2006). The translator’s task was 
defined as preparing a translation that accorded with the rules of the target language as 
well as the country context and ensuring that the translation had the same meaning as 
the source text. 

'The major guidelines to assess the quality of translation were: 

• Translations should have the same register (language level, degree of formality) as 
the source text; 

• Translated passages should employ correct grammar and usage (for example subject/ 
verb agreement, prepositions, verb tenses); 

• Translated passages should neither clarify, omit, nor add information; 

• Translated passages should employ equivalent qualifiers and modifiers, in the order 
appropriate for the target language; 

• Idiomatic expressions should be translated appropriately, not necessarily word for 
word; and 

• Spelling, punctuation, and capitalization in the target text should be appropriate for 
the target language and the country/ cultural context. 

In general, the translators were asked to pay particular attention to the following: 

• Finding words and phrases in the target language that were equivalent to those in the 
international version; 

• Making sure that the essential meaning of the text did not change; 

• Making sure that the translated questionnaires asked the same questions as those 
in the international version and that national adaptations were made appropriately; 
and 

• Being aware of possible changes in the layout of the instruments due to translation. 
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For countries administering the SITES 2006 questionnaires in English, these 
guidelines were applicable to any changes made to adapt the international version to the 
national context. 

A limited number of national adaptations were required for the SITES 2006 
questionnaires. These included a national school definition, the target grade name, and 
the percentage of students who were native speakers of the language of instruction. 
Information that had to be replaced with the nationally appropriate term on a mandatory 
basis was presented in carets (< >). 

Participating systems could remove questions or options not applicable to their 
particular system. If considered necessary, they could also add possible national questions 
and additional categories. All changes had to be thoroughly documented and, in the case 
of additional categories, recording instructions were required. 

The NRCs were cautioned that changes made when preparing the national 
version of the instruments would potentially increase the likelihood of errors and might 
eventually cause loss of data in the international database. 

5.2.5 Documenting National Adaptations 

NRCs were required to use “national adaptation forms” to document all adaptations 
made to the international questionnaires. The forms had to be completed and reviewed 
at various stages of the preparation of national instruments. Version I was completed 
during the internal translation/adaptation and review process and sent (together with 
the translated/ adapted questionnaires) for translation verification (see below). After the 
language verification, the NRCs updated the forms to reflect any changes resulting from 
it and sent Version II as well as the instruments destined for layout verification to the 
ICC (see below). Version III, the final version of the questionnaires, was sent to the lEA 
DPC as a documentation of national adaptations (see Chapter 9). 

The national adaptation forms, supplied as electronic documents, included detailed 
instructions on how to complete the form at each stage of the national instruments 
preparation. Appendix D provides a list of adaptations made by study participants. 

5.3 International Translation/Adaptation Verification 

Once the survey questionnaires had been translated, adapted, and reviewed, they and the 
national adaptation forms were submitted to the lEA Secretariat for language verification 
by the independent language specialist selected in cooperation with Lionbridge Inc. 

The international language verifiers for SITES 2006 were required to have the 
target language as their first language, to have formal credentials as translators working in 
English, to b e educated at university level, and (if possible) to live and work in the country 
for which the verification was carried out (or be in close contact with this country). 

5.5.7 Translation Verification Process 

The international translation verifiers received general information about the study and 
the design of the instruments together with a description of the translation procedures 
used by the national centers. They also received detailed instructions for reviewing the 
instruments and registering deviations from their original version. 

The primary task of the language verifiers was to evaluate the accuracy of the 
translation and the adequacy of the national adaptations (reported in the national 
adaptation forms). The instructions given to verifiers emphasized the importance of 
maintaining the meaning and complexity level of the questions included in each of the 
questionnaires. Specifically, verifiers had to ensure the following: 

• The translation had not affected the meaning or difficulty of the text; 

• The questions had not been made simpler or more complex when translated/ 
adapted; 
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• No information had been omitted from or added to the translated text; 

• The questionnaires contained all correct questions and answer options; and 

• The national adaptation forms reflected all adaptations incorporated into the national 
test instruments. 

The verifiers documented any errors or suggested changes directly in the submitted 
Microsoft® Word documents, using the “Track Changes” and “Insert Comments” features 
of the application. Verifiers were also asked to suggest — if necessary — an alternative that 
would improve the comparability (i.e., the equivalence between the adapted version and 
the international source version) and to provide an overall evaluation of the translation, 
its accuracy, and its cultural relevance. 

To help NRCs understand the comparability of the translated text with the 
international version, verifiers were asked to assign a “severity code” to any deviations. 
The codes, which ranged from 1 (major change or error) to 4 (acceptable change), signaled 
the following: 

1 — major change or error: Examples included incorrect order of choices, omission of 
question or response option, incorrect translation resulting in the answer being 
suggested by the question, an incorrect translation that changed the meaning or level 
of complexity of the question, and incorrect order of the questions. 

2 — minor change or error: Examples included spelling errors that did not affect 
comprehension, misalignment of margins or tabulations, inappropriate changes in 
font or font sizes, and discrepancies in the headers and footers of the document. 

3 — suggestion for alternative: The translation may have been adequate, but the verifier 
suggested a different wording. 

4 — acceptable change: The change was acceptable and appropriate but was not 
documented in the national adaptation forms. 

The translation/adaptation verification feedback was sent to the national centers 
so that the NRCs could review the suggestions of the translation verifiers and revise 
the instruments accordingly. The NRCs were also asked to complete the “translation 
verification summary form,” which required them to comment on those suggestions 
made by verifiers that they decided not to implement even if those suggestions had been 
marked as a major change or error (severity code l). 

5.5.2 Results of the Translation/Adaptation Verification 

Although the verification of translations and national adaptations produced a good or very 
good quality of the national versions prepared by the SITES 2006 participants, verifiers 
of all languages found errors and suggested improvements. The errors included: 

• Translation: mistranslations, inaccurate translations, “word byword” translations 

• National adaptations: improper terminology, inconsistencies in adaptations 

• Punctuation and capitalization: not appropriate for the target language 

• Grammar: use of English sentence structure inappropriate for target language 

• Missing words and typographical errors. 

The NRCs were expected to review the feedback they received from the 
international verifiers and to implement changes if they accepted the suggestions. With a 
few exceptions, the NRCs accepted all or almost all such suggestions. The major objects 
of disagreement were some adaptations (especially terminology used in the educational 
settings), proposed synonyms, language register, and use of foreign (English) terms. 
Rejected suggestions were documented so they could be used during interpretation of 
unusual results, and thereby determine if the results could be explained in terms of errors 
in the translation or adaptation of the survey instruments. 
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5.4 International Layout Verification 

After translation, adaptation, and verification, and before the participants could use the 
national version of the international instruments, one last quality assurance step — layout 
verification — took place at the University of Twente, the Netherlands. This step involved 
reviewing each questionnaire in its print-ready form. The ICC compared each of the 
translated and adapted questionnaires with the international version, documenting 
discrepancies between the two. The checks consisted of: 

• Appropriate question numbering 

• Same number of answer options in the national and international versions 

• Consistent formatting (fonts, bold/italics, indenting, etc.) 

• Existence of answer instructions in each question 

• Appropriate referencing and skip instructions after filter questions 

• Availability of return instructions 

• Appropriate grade (range) referencing. 

Any deviations discovered were communicated to the NRCs, and revised versions 
of the instruments were re-checked until all problems were solved satisfactorily. Overall, 
this phase of quality control proved to be very useful. Although there were a few countries 
where, during the first step of layout verification, no deviations were discovered, in many 
countries revisions were needed. The seriousness of deviations that were discovered 
ranged from relatively minor to very serious and can be summarized as follows: 

• Minor deviations, such as: 

- Missing periods (and/ or question marks) at the end of sentences 

- Answer alternatives not lining up consistently 

- Indenting of text not optimal 

- Too much space between stem of question and answer instruction 

- Missing lines in a table. 

• Major deviations, such as: 

- Page breaks in questions 

- Empty pages 

- Inconsistent font sizes 

- Answer boxes too small 

- Yellow highlights (put in place to signal NRCs to pay special attention) not 
removed 

- Wrong numbering of questions 

- Wrong numbering of answer options 

- No or incorrect translation of footers 

- Certain words not translated (e.g., “NA” as abbreviation of “Not Applicable”) 

- Inconsistent answer instructions 

- Missing date for return of questionnaires 

• Serious deviations (which would lead to loss of data): 

- Answer options missing 

- Missing or misleading answer instructions 

- Incorrect referencing in filter questions 

- Incorrect grade-range or target-grade specifications. 

Irrespective of the seriousness of the deviations, the NRCs revised the instruments 
until all problems were solved. In most cases, they completed this work in a few days, but in 
some cases more iterations ( 3 -I-) were needed to resolve inconsistencies and deviations. 

Reference 

SITES 2006 International Coordinating Center (2006). SITES 2006 survey operations manual. 
Hamburg: lEAData Processing and Research Center (DPC). 
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6.1 Overview 

A key element of the quality of any international comparative study is the selection of 
quality samples. Only properly selected and representative samples can yield unbiased, 
accurate, and internationally comparable survey estimates. SITES 2006 collected data at 
two levels: 

• The school level, through (i) a school principal questionnaire, and (ii) a ICT 
coordinator questionnaire 

• The classroom level, through a teacher questionnaire. 

Student achievement surveys such as the Progress in International Reading 
Literacy Study (PIRLS) and the Trends in International Mathematics and Science 
Study (TIMSS) also collect information at the school level and at the teacher level, but 
mainly or exclusively for explaining the variability of student performance. In other 
words, such surveys are primarily interested in the student population, which means 
the teacher population cannot be modeled directly. In contrast, the SITES research 
objectives imply that data and results are reported at the school level and at the teacher 
level. Therefore, it defines two target populations: (i) the school population, and (ii) the 
teacher population. 

The sampling design had to be optimized to ensure accuracy of the survey 
estimates at both levels. A sampling design that would have led to sacrificing the accuracy 
of the estimates at one level for the accuracy of the estimates at the other level would have 
been incompatible with the project’s purposes. 

This chapter describes the sampling design of the SITES study. It consists of (a) 
the definition of the target populations, (b) the school and within-school exclusions, and 
(c) the sampling design. Sampling in SITES 2006 was a primary responsibility of the 
lEA Data Processing and Research Center (DPC) in Hamburg, Germany. 

6.2 Target Population Definitions 

6.2.1 School Population 

The international-desired school target population is defined as all schools 
where students are enrolled in the target grade, that is, in the grade that represents 
eight years of schooling, counting from the first year of ISCED Level 1. 

In most education systems. Grade 8 corresponds to the eighth grade of compulsory 
education. Compulsory education usually starts with the first year of Level 1 of the 
International Standard Classification of Education (ISCED) (UNESCO Institute for 
Statistics, 2006). 
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The definition of the desired target school population for SITES 2006 differs 
slightly from the desired target population implemented during the school data collection 
for SITES-Ml (Pelgrum & Anderson, 1999). However, in most systems, the 2006 target 
grade was similar to the target grade used for the Module 1 data collection. It was also 
consistent with the definition of the TIMSS desired target population, and therefore it is 
possible to link, at the system level, the results of both surveys. 

All schools of all educational sub-systems that involved students learning full-time 
and part-time in the appropriate target grade were part of the international- desired target 
population. Schools that did not contain the target grade were, by definition, excluded 
from the study. In simple terms, the international-desired school target population was 
designed to provide full coverage of schools attended by target grade students in an 
education system. 

6.2.2 Teacher Population 

The SITES 2006 mathematics teacher target population is defined as all teachers 
ofi mathematics teaehingin the target grade (i.e., in the grade that represents eight years 
ofi schooling, countingfirom the first year of ISCED Level l) in the school year in which 
the survey is conducted. 

The SITES 2006 science teacher target population is defined as all teachers teaching 
science (or, depending on the education systems, teaching biology, physics, chemistry, 
and earth science, if appropriate) in the target grade (i.e., in the grade that represents 
eight years ofi schooling, countingfirom the first year of ISCED Level l) in the school 
year in which the survey is conducted. 

This definition implies the exclusion (from the international- desired target 
population) of teachers who may have taught in the target grade in previous school 
years (but not in the school year of the SITES 2006 survey), or teachers who might have 
taught in the target grade in subsequent school years (but not in the school year of the 
SITES survey). 

The research team considered it necessary to separate the mathematics teacher 
and the science teacher populations, because of the likelihood of some teachers teaching 
both subjects in a school. 

6.3 Exclusions 

The final sampling design contained the following concepts: 

• The international-desired population; 

• The national-desired population, that is, the international-desired target population 
minus exclusions from the national coverage (see Section 6.3.1); 

• The national-defined population, that is, the national-desired target population minus 
school-level exclusions (see Section 6.3.2); and 

• The national- effective population, i.e., the national- defined population minus the 
within-school exclusions (see Section 6.3.3). 

Figure 6.1 illustrates these different concepts. 
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Figure 6.1: International and National-desired Populations, National-defined Population, 
and Exclusions 




By definition^ there is no difference between the national-defined target population 
and the national-effective target population of schools. 

6.3.1 Exclusion from the National Coverage 

The participating education systems were encouraged to provide complete national 
coverage in their national-desired target population. However, the research team 
recognized that political, organizational, and/or operational reasons could make it 
extremely difficult for some systems to meet this objective. Consequently, systems 
were allowed to reduce their national coverage by removing a geographical region, an 
educational sub-system, or even a language group. 

In case of reduced coverage, the national-desired target population differed from 
the international-desired target population and therefore no longer represented the 
entire national school system. Appendix B reports exclusions from the national- desired 
target populations per system. 

6.3.2 School Exclusions 

Schools were excluded for the following reasons: 

• They were teaching only mentally or functionally disabled students; 

• They were geographically inaccessible; 

• They were extremely small in size; or 

• They offered a curriculum, or school structure, radically different from the mainstream 
education system. 

School exclusion categories are reported per system in Appendix B. 

6.3.3 Within-school Exclusions 

Instances where teachers were teaching only mentally or functionally disabled students 
required a procedure for excluding these teachers within a sampled school. Thirteen 
education systems used this option. For example, because of the special structure of the 
education system in Finland, many of the country’s lower secondary school teachers 
teach exclusively to special education students. As a result, 7.3% of Finland’s mathematics 
teachers and 4.1% of the country’s science teachers were excluded. The percentage of 
excluded teachers within schools in the other systems ranged from 0.1% to 2.0%. 




SITES 2006 TECHNICAL REPORT 



Table 6. 1 gives the exclusion rates for schools and the un-weighted exclusion rates 
for the mathematics and the science teachers within these schools. 

Table 6.1 : Exclusion Rates in SITES 2006 



Education System 


School-level Exclusions 
(% of Students) 


Within-school Exclusions 
(% of Mathematics Teachers) 


Within-school Exclusions 
(% of Science Teachers) 


Alberta, Canada 


1-4 


1.0 


1.0 


Catalonia, Spain 


0.6 


0.2 


0.1 


Chile 


0.0 


0.2 


0.0 


Chinese Taipei 


0.9 


2.0 


0.8 


Denmark 


1-3 


0.9 


0.8 


Estonia 


3-1 


0.0 


0.1 


Finland 


2-4 


7-3 


4-1 


France 


1-5 


0.1 


0.0 


Hong Kong SAR 


3-6 


0.0 


0.0 


Israel 


n .3 


2.0 


1.0 


Italy 


0.0 


0.0 


0.0 


Japan 


0.0 


0.0 


0.0 


Lithuania 


2.2 


0.0 


0.0 


Moscow, Russian Federation 


0.2 


0.5 


0.1 


Norway 


0.7 


1.2 


0.0 


Ontario, Canada 


3-9 


1.2 


0.6 


Russian Federation 


1.1 


0.0 


0.0 


Singapore 


0.0 


0.0 


0.0 


Slovak Republic 


0.0 


1-4 


0-5 


Slovenia 


0.2 


0.0 


0.0 


South Africa 


0.0 


0.0 


0.0 


Thailand 


4.6 


0.0 


0.0 



6.4 School Sample Design 

International surveys in education usually proceed in two steps. First, a sample of schools 
is selected from a complete list of schools containing the population of interest. Then, 
a simple random sample of students, classes, or teachers is drawn within the selected 
schools. In the literature, such sampling design is usually called two-stage sampling. 
Broadly speaking, schools can be selected with equal or unequal probabilities. Both 
options were discussed during the development of the SITES 2006 sampling design. 

Selecting schools with equal probability would have given every school the same 
chance to be included in the survey. However, because SITES 2006 selected a small 
number of teachers within sampled schools, selecting schools with equal probability 
would have generated a large variability in the selection probability of the teachers. 
Within a sampled school, the probability of selection of a teacher at a school with 20 
mathematics teachers would be five times smaller than the probability of selection of a 
teacher in a school with four mathematics teachers. Such variability in the probabilities 
of selection generally results in less accurate estimates of statistics for the intended target 
population. 

Use of a probability proportional to size (PPS) design leads to a school with 20 
teachers being five times more likely to be selected than a school with four teachers. 
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But within a sampled school, the prohahility of a teacher being selected is inversely 
proportional to the number of teachers. Therefore, selecting schools with probabilities 
proportional to their size and selecting a fixed number of teachers with equal probabilities 
within sampled schools minimizes the variability of the total selection probability of the 
teachers. 

Thus, a PPS design would be appropriate for SITES if the project focused on the 
teacher population alone. However, because SITES collected and reports data at two 
levels — at the school level and at the teacher level — a PPS sample design would have 
generated a large variability of the school selection probability and, consequently, a large, 
and undesirable, variability of the school weights. 

To meet the conflicting requirements of a school survey and a teacher survey, the 
SITES 2006 research team implemented a sample design that involved the following: 

• Stratifying the school sample frame according to the school size; 

• Selecting, within an explicit school-size stratum, schools with an equal probability of 
selection; and 

• Selecting, within sample schools, teachers with an equal probability of selection. 

The following sections describe in more detail the sampling procedures 
implemented for the school sample and for the teacher samples. 

6.4.1 Explicit Stratification 

Before sampling takes place, schools are stratified within the sampling frame. Stratification 
consists of grouping schools into strata according to some grouping, or stratification, 
variables. Stratification is generally used for the following reasons: 

• To improve the efficiency of the sample design, thereby making survey estimates 
more reliable; 

• To apply different sample designs, such as disproportionate sample allocations, to 
specific groups of schools, such as states or provinces; and 

• To ensure representation of specific groups within the target population in the 
sample. 

Examples of stratification variables include: 

• School size (large, medium, small) 

• Regions (states, provinces) 

• Urbanization (rural areas, urban areas) 

• Socioeconomic status (low, medium, high) 

• School types (public, private) 

• School programs (primary, elementary, secondary). 

Appendices B and C list the explicit stratification variables used in the participating 
school systems. 

6.4.2 Size Stratification 

As mentioned above, school size was the most important explicit stratification variable 
for SITES 2006. Size stratification is therefore always used as the last level of explicit 
stratification. Thus, with SITES, size strata were formed within each of the other explicit 
strata that the NRCs proposed. 

The idea behind having size strata is to avoid grouping schools that overly differ in 
size in the same stratum. This approach allowed the SITES research team to better control 
the variability of school and teacher weights and consequently to achieve more reliable 
estimates. Because teachers within large schools have a lower probability of selection than 
teachers within small schools, the combined teacher probability of selection is smaller in 
large schools than is the combined teacher probability of selection within small schools. 
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This variability of selection is also reflected in the teacher weights. Grouping 
schools in size strata makes it possible to select: 

• More large schools, thus increasing their probability of selection; and 

• Fewer small schools, thus decreasing their probability of selection. 

The difference in selection probability between large schools and small schools 
provides partial compensation for the variability associated with selection probability of 
teachers within schools. 

The size stratification method was developed by Marc Joncas of Statistics Canada, 
Ottawa (personal communication). 

For SITES 2006, the formation of size strata consisted of four steps, described in 
Sections 6.4.2. 1 to 6.4.2.4. Section 6.4.4 provides an example of the stabilization of the 
weights. 

6.4.2. 1 Step 1: School Sample Size Allocation per Explicit Strata 

To create size strata, one needs a measure of size (MOS) for each school in the school 
sampling frame. For SITES 2006, optimal MOS would have been the number of 
mathematics teachers and science teachers teaching in the target grade during the 
school year of the survey. However, because this situation is never or rarely possible, 
the systems participating in the study were encouraged to use as MOS the number of 
students enrolled in the target grade. This MOS gives an indication of the number of 
eligible teachers in each listed school. 

Possible school MOS, in decreasing order of suitability, were: 

• Student enrolment in the target grade; 

• Average student enrolment per grade, that is, total student enrolment divided by the 
number of grades in the school; 

• Number of classrooms in the target grade; and 

• Total student enrolment. 

Current enrolment data, however, are rarely available at the time schools are 
sampled. Therefore, with SITES 2006, the MOS in the school sample frame represented 
the school enrolment of a previous school year. The quality of a sampling frame depends, 
to a large extent, on the accuracy of the MOS available. Systems were therefore encouraged 
to use the latest data available. 

If a national center did not require over- sampling, the school sample size per 
explicit stratum was equal to: 

_ rp I ^ MOSj \ 

2 \N ^ MOS I' 
where 

• N is the number of schools in the system, and JV; is the number of schools in the 
explicit stratum i; 

• n is the number of schools in the sample and n; is the number of schools to be sampled 
in the explicit stratum j; and 

• MOS is the total number of students in the target grade across schools in the system 
and MOSi is the total number of students in the target grade across schools in the 
explicit stratum i (or any other measure of size used by the system). 

As can be seen, the formula is a compromise between an allocation that is 
proportional to the number of schools and an allocation that is proportional to the 
number of students (and thus probably for teachers). 

6. 4.2.2 Step 2: Definition of the Size Strata per Explicit Stratum 

Size strata need to be created within each explicit stratum. The number of size strata per 
explicit stratum depends on n, , that is, the number of schools available to sample in the 
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explicit stratum. The following algorithm describes how the number of size strata was 
determined: 

• If up to 14 schools were to be sampled in the explicit stratum, no size strata were 
created; 

• If between 15 and 24 schools were to be sampled in the explicit stratum, two size 
strata were created; 

• If between 25 and 34 schools were to be sampled in the explicit stratum, three size 
strata were created; 

• If between 35 and 44 schools were to be sampled in the explicit stratum, four size 
strata were created; 

• If more than 45 schools were to be sampled, five size strata were created. 

6.4.23 Step 3; Allocation of Schools per Size Stratum within an Explicit Stratum 
Schools in the explicit stratum were allocated to their respective size strata. Within these 
size strata, schools were sorted according to their implicit stratification variable and by 
their measure of size (MOS), that is, in most systems, the number of students in the 
target grade. 

6.4.2.4 Step 4: School Sample Size Allocation per Size Strata 

The number of schools to be sampled within each size stratum is equal to: 

2\Ni MOSij’ 

where 

• Ni is the number of schools in the explicit stratum i, and Nij is the number of schools 
in the size stratum; within the explicit stratum i; 

• ni is the number of sampled schools in the explicit stratum i, and is the number of 
sampled schools in the size stratum; within the explicit stratum i; and 

• MOSi is the total number of students in the target grade across schools in the explicit 
stratum i, and MOSij is the total number of students in the target grade across schools 
in the size stratum; within the explicit stratum i. 

6.4.3 Implicit Stratification 

Implicit stratification consists of sorting the school sampling frame by a set of implicit 
stratification variables. This type of stratification, which was very effective for the school 
sample selection method implemented in SITES 2006, is a very simple way of ensuring a 
strictly proportional sample allocation of schools across all implicit strata. It can also lead 
to improved reliability of survey estimates, provided the implicit stratification variables 
being considered are known to have a significant between-strata variance component. 
Finally, as will be discussed later, implicit stratification ensures that replacement schools 
are similar in some respect to sampled schools. 

Appendices B and C set out the implicit stratification variables used in the 
participating school systems. 

6.4.4 Exam pie for Stabilization of Weights due to Size Stratification 

The following student-level example demonstrates the efficiency with which the sampling 
design minimizes the weight variability. Let us suppose that, in a fictitious study about 
students, a system decided to group schools within the four following explicit strata: 
(i) private rural, (ii) private urban, (iii) public rural, and (iv) public urban. Based on 
this system’s school frame, 100 schools need to be sampled and 35 students per sampled 
school selected. Table 6.2 presents the number of schools per explicit stratum, their 
respective number of students, the school sample size based on the above algorithm, and 
the school and student probabilities of selection. 



SITES 2006 TECHNICAL REPORT 



Table 6.2: School Allocation and Probabilities of Selection According to the SITES Design 



Explicit 

Stratum 


Number of 
Schools 


Number of 
Students 


School 
Sample Size 


School 

Selection 

Probability 


Average Student 
Probability 
within School 


Combined 

Student 

Probability 


Private rural 


TOO 


10,000 


175 


0.175 


0.350 


0.061 


Private urban 


TOO 


20,000 


22.5 


0.225 


0.175 


0.039 


Public rural 


TOO 


30,000 


275 


0.275 


0.117 


0.032 


Public urban 


TOO 


40,000 


32-5 


0.325 


0.088 


0.028 



The ratio between the largest and the smallest school probabilities of selection 
is equal to 0.325 divided by 0.175, that is, 1.86. Similarly, the ratio between the largest 
and the smallest combined student probabilities of selection is equal to 0.061 divided 
by 0.028, that is, 2.15. Table 6.3 presents the same results in the case of a proportional 
allocation to the number of schools. Table 6.4 presents the same results in the case of a 
proportional allocation to the number of students. 



Table 6.3: School Allocation and Probabilities of Selection According to a Proportional 
Allocation to the Number of Schools 



Explicit 

Stratum 


Number of 
Schools 


Number of 
Students 


School 
Sample Size 


School 

Selection 

Probability 


Average Student 
Probability 
within School 


Combined 

Student 

Probability 


Private rural 


100 


10,000 


25 


0.25 


0.350 


0.088 


Private urban 


100 


20,000 


25 


0.25 


0.175 


0.044 


Public rural 


100 


30,000 


25 


0.25 


0.117 


0.029 


Public urban 


100 


40,000 


25 


0.25 


0.088 


0.022 



Table 6.4: School Allocation and Probabilities of Selection According to a Proportional 
Allocation to the Number of Students 



Explicit 

Stratum 


Number of 
Schools 


Number of 
Students 


School 
Sample Size 


School 

Selection 

Probability 


Average Student 
Probability 
within School 


Combined 

Student 

Probability 


Private rural 


100 


10,000 


10 


0.1 


0.350 


0.035 


Private urban 


100 


20,000 


20 


0.2 


0.175 


0.035 


Public rural 


100 


30,000 


30 


0-3 


0.117 


0.035 



The proportional allocation to the number of schools guarantees no variability of 
probabilities at the school level but generates a larger variability at the student level (the 
ratio is equal to 0.088/ 0.022, i.e., 4). Inversely, the proportional allocation to the number 
of students guarantees no variability of probabilities at the student level but generates a 
larger variability at the school level (the ratio is equal to 0.4/ 0. 1, i.e., 4.). 

Because the number of teachers per schools is expected to closely correlate with 
the number of students per school, this example clearly illustrates the compromise used 
in the SITES 2006 sampling design to minimize the variability of the probabilities and 
weights at the school level and at the teacher level. 
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6.5 School Sample Selection 

The sample selection method used was systematic sampling with equal selection 
probability within each size stratum. In the sampling frame, the schools in each size 
stratum were listed in order of the implicit stratification variables. Within the implicit 
strata, the schools were sorted by school size. After this, each school had a fixed rank 
number in the sampling frame. A sampling interval was then computed within each size 
stratum. This sampling interval was equal to: 



where 

Ny is the number of schools in the size stratum; within the explicit stratum i ; and My is 
the number of sampled schools in the size stratum; within the explicit stratum i. 

The first school was sampled by choosing a random number between 0 and SI,;, 
the sampling interval. The school whose rank in the list was equal to or larger than this 
number was sampled. By adding the sampling interval to the first random number, the 
second school was identified in the same way. This process of repeatedly adding the 
sampling interval to the previous selection number resulted in a school sample of the 
required size. 

The default school sample size in SITES 2006 was 400 schools per participating 
system. In systems with many small schools, the sample size had to be increased to ensure 
a teacher sample size large enough to allow all the data analyses that the SITES 2006 
consortium intended to perform. 

6.6 Replacement Schools 

With any survey in education, it is not always possible to obtain the participation of 
all sampled schools. Avoiding the resulting sample size losses requires a mechanism to 
identify a priori replacement schools for non-participating sampled schools. Another 
perhaps more important reason for identifying replacement schools a priori is to avoid 
the haphazard use of alternate schools as replacements, which may amplify response 
biases. Although this approach does not necessarily avoid non-response bias, it does 
appear to minimize the potential for bias. Furthermore, this approach is conceptually a 
more elegant means of accommodating a low participation rate than is over-sampling. 

With SITES 2006, each sampled school in the main study was assigned two 
replacement schools in the sampling frame according to the following algorithm: the 
school immediately following the sampled school was the first replacement school of 
the sampled school and the school immediately preceding any sampled school was the 
second replacement school of the sampled school. The use of implicit stratification 
variables, and the subsequent ordering of the school sampling frame by size, ensured 
that any sampled school’s replacements had stratification characteristics similar to those 
of the sampled school. 

However, in small systems, replacement schools could not be identified for all 
sampled schools. Therefore, some sampled schools that refused to participate were not 
replaced. 

6.7 Teacher Sampling Design 

Sampled and participating schools had to complete two “teacher listing forms,” one for the 
mathematics teachers and one for the science teachers. All teachers from the appropriate 
target grade in a sampled school had to be recorded on these forms. In schools where 
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science was being taught as individual subjects (i.e., biology, physics, chemistry, earth 
science), any teacher teaching any one of these subjects was listed only once on the 
science teacher listing form. For example, if physics, biology, and earth science were being 
taught in a school, all the teachers of the three subjects were entered on the same list. If 
a teacher was teaching both physics and biology, he or she was entered only once on the 
science teacher listingform. However, the fact that he or she was teaching two subjects was 
also recorded in the listing on this form. Finally, a teacher teaching both mathematics 
and science in the target grade was entered on both teacher listing forms; the teacher 
identification numbers on both forms made it possible to link one form to the other. 

With the teacher listing forms, the school coordinators were also requested to 
estimate the percentage of mathematics teachers and science teachers that might be 
considered as ICT-using teachers. ICT-using teachers were defined as teachers who use 
computers (or an equivalent) at least once a year with their students for teaching/learning 
purposes. “Equivalent” as used here means machines offering the same functionalities as 
a computer, for example, some personal digital assistants (PDAs), terminals connected 
to a server, and so on. 

The data from the teacher listing forms, once returned to the national center, were 
recorded in the within-school sampling software WinW3S (see Section 7.3) provided by 
the lEA DPC. Completion of this work was followed by selection of a teacher sample. 

The first step of the selection process consisted of determining the number of 
teachers to be sampled in a particular school. The teacher sample size related to the 
percentage of ICT users. Thus: 

• Two teachers per subject for any school with an estimated percentage between 76 
and 100; 

• Three teachers per subject for any school with an estimated percentage between 51 
and 75; 

• Four teachers per subject for any school with an estimated percentage between 1 and 
50; and 

• Two teachers per subject for any school with an estimated percentage of 0. 

These within-school sample sizes were considered as minimal requirements 
unless the sampled school counted fewer than two or three teachers, depending on 
the percentage of ICT-using teachers. Systems were, however, allowed to sample more 
teachers than specified under these minimal requirements. 

The next step was to select the teachers according to a systematic random sample 
procedure. 

In a large number of the systems participating in the SITES 2006 survey, many 
teachers were teaching both mathematics and science. These teachers are referred to 
from here on as “double subject teachers.” 

For the field trial, the mathematics teacher sample and the science teacher sample 
were drawn independently within schools. For the main study, the research team decided 
to link the random numbers of both teacher listing forms in order to reduce the number 
of teachers being selected twice. 

The following example (see also Figure 6.2) illustrates this approach. Let us 
suppose that a particular school participating in SITES 2006 had eight mathematics 
teachers and eight science teachers, and of these 16 teachers, four were teaching both 
subjects. In this instance, it would be vitally important for the teachers teaching both 
subjects to be listed on the top of both lists, after which the teachers only teaching one 
subject would be listed. It would also be important in this situation to guarantee that the 
order of the double subject teachers is identical in both teacher lists. (These two aims 
were achieved in SITES 2006 through automatic sorting prior to sampling within the 
WinW3S software.) 



SAMPLING DESIGN 



To continue the example: the sampling interval for hoth lists is equal to 8/4=2. 
Supposing that the random number for mathematics was O.2., then, as described above, 
the sampled mathematics teachers would be the teachers with numbers 1, 3, 5, and 7. 
Because the school had combined teachers, the procedure links the random numbers as 
follows: 

• Where the random number for the mathematics list is lower than or equal to 0.5, the 
random number for the science list is the mathematics random number plus 0.5; 

• Where the random number for the mathematics list is higher than 0.5, the random 
number for the science list is the mathematics random number minus 0.5. 

In the above example, the random number for mathematics is 0.2, so the random 
number is equal to 0.2 + 0.5 = 0.7. The sampled teachers for science would then be: 

. (0.7*2) + l = 2.4,thatis,2 

• 2.4 + 2- 4.4, that is, 4 

• 4.4 + 2- 6.4, that is, 6 
6.4 + 2 = 8.4, that is, 8. 

As can be seen, the outcome is no overlapping for the double subject teachers. 



Figure 6.2: Example of the Within-school Sampling Procedure 



Mathematics List 


Science List 


1 


Teacher A 


1 


Teacher A 


2 


Teacher B 


2 


Teacher B 


3 


Teacher C 


3 


Teacher C 


4 


Teacher D 


4 


Teacher D 


5 


Teacher E 






6 


Teacher F 






7 


Teacher G 






8 


Teacher H 










5 


Teacher 1 






6 


Teacher J 






7 


Teacher K 






8 


Teacher L 



However, the efficiency of the procedure can be reduced if the sampling interval is 
not identical for both lists. In this situation, it is still possible for a teacher to be selected 
for both the mathematics sample and the science sample. When this occurs, and only 
when this occurs, a double sampled teacher is replaced by a double subject teacher not 
selected for both samples. If no replacement teachers are identified, the double sampled 
teachers are invited to fill in the mathematics and the science teacher questionnaires. 
This was the procedure followed for SITES 2006. 

Finally, the teacher questionnaire explicitly asked the teacher to refer to a particular 
class, identified as the “target class,” when answering some of the questions. However, 
there were teachers who were teaching more than one classroom in the target grade. 
Allowing teachers to self-select which class to refer to would have introduced a bias of 
unknown magnitude, as the teachers could have, for example, selected the class in which 
they most often used ICT. The classes taught at the target grade by a particular teacher 
were therefore listed on the teacher listing forms. It was subsequently important that the 
NRCs randomly selected the class the teacher had to refer to. In practice, the random 
selection of the target class was performed by WinW3S, the lEA DEC software designed 
to sample teachers and target classes within sampled schools. 
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6.8 Quality Control 

The lEA DPC selected nearly all of the school samples for SITES 2006. The only school 
sample to be selected by a local SITES team was the school sample for Japan. The sample 
was then verified at the DPC and found to be correct and acceptable. 

The NRCs were responsible for providing a complete and comprehensive school 
sampling frame with a suitable measure of size. The suitability of this frame was checked 
by systematic comparison of the data with frames of preceding surveys as well as with 
census data. 

The details of the national sampling plans were discussed with NRCs. Every 
attempt was made to implement the best possible sample design, particularly with respect 
to the most efficient method of stratification. Any deviations from the international 
sample design had to be approved by the sampling referee. 

After sample selection, the school samples underwent a systematic checking 
procedure. Each sampling step was thoroughly documented so that the randomness of 
the selection could be proven. This documentation was sent to the NRCs for review. 

Within-school sampling was performed using the WinW3S software, which 
ensured that all within-school sampling quality criteria were enforced. Any deviation 
detected from the within-school sampling procedures led to annotation of the results in 
the international report. 
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Survey Operations and Procedures 

Falk Brese 
Ralph Carstens 



7.1 Overview 

The SITES 2006 data collection was carried out locally by the participating education 
systems once they received standards, guidelines, and detailed procedures for all survey 
activities from the lEA SITES consortium. Data collection was a demanding exercise 
involving administration of school and teacher questionnaires in, on average, 400 
schools per education system. Conducting a successful data collection called for close 
cooperation between the international consortium and experts, the NRCs, the within- 
school liaison personnel (referred to as “school coordinators”) and, eventually, the 
respondents. 

The SITES 2006 survey operations procedures were derived from procedures 
successfully applied in TIMSS 2003, PIRLS 2006, and other lEA studies. However, 
due to the specific target population and the aims of the study, the survey operations 
in SITES 2006 differed considerably from those employed in student-level studies. 
Most importantly, SITES 2006 targeted adult populations of school principals, ICT 
coordinators, and teachers. Additionally, the administration was neither organized in 
fixed-time sessions nor supervised by external administrators. The schools selected for 
SITES and the designated respondents within the schools — for the most part, teachers — 
typically received either the paper questionnaires or invitations to access the online 
version (see Chapter 8) via standard mail. This situation made securing the participation 
of sampled and designated individuals and monitoring progress particularly challenging, 
and meant that adequate procedures and approaches needed to be in place to prevent 
problems. 

This chapter describes the survey operations for collecting the data. It includes the 
responsibilities of the NRCs and school coordinators, the procedure for sampling and 
tracking teachers within schools, and the steps involved in administering the principal, 
technical, and teacher questionnaires and preparing the materials for data entry. 

7.2 National Research Coordinator and School Coordinator 
Responsibilities 

The NRC was the key person with ultimate responsibility for collecting data 
for the SITES 2006 survey according to internationally agreed procedures and 
specifications. NRC responsibilities in other areas were outlined in earlier chapters 
of this report. The following sections focus on administration of the survey in 
participating schools. Specifically, they describe the procedures for sampling and 
tracking teachers within sampled schools, and for organizing the administration of the 
questionnaires. 
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On receiving the school sample information from the sampling team at the lEA 
DPC (see Chapter 6), the NRCs began contacting the designated schools to secure their 
participation. If a sampled school was unahle or unwilling to participate in the survey, 
the NRC concerned contacted the first replacement school and asked for participation. 
If this school declined participation, the NRC contacted the second replacement school. 
After agreeing to participate, each school was asked to identify a “school coordinator” — a 
person who would remain in contact with the national center and carry out all SITES- 
2006-related school-level tasks. The coordinator was usually a teacher or other school 
staff member, who ideally was not participating in the study, or he or she was an outsider 
appointed by the NRC. In either case, the NRC was responsible for ensuring that the 
school coordinator became familiar with their responsibilities as detailed in the SITES 
2006 School Coordinator Manual (lEADPC, 2006c). 

Figure 7.1 presents the major activities completed at the national center under 
the supervision of the NRC and at the school by the school coordinator before and 
after survey administration. These activities were supported by the WinW3S software, 
which automatically produced all necessary forms, lists, and labels, and assisted NRCs in 
keeping track of the status of their survey as described in the next sections. 

Figure 7.1: NRC and School Coordinator Activities 
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7.3 Operational Manuals and Software 

NRCs received the following set of procedural manuals detailing all aspects of the data 
collection: 

• The SITES 2006 Sampling Manual (lEA DPQ 2006a) defined the SITES 2006 target 
populations and sampling goals and described the procedures for the sampling of 
schools. 

• The SITES 2006 Survey Operations Manual (lEA DPQ 2006b) was the essential 
handbook for the NRC. It described in detail all of his or her expected activities and 
responsibilities, from the moment the SITES international instruments arrived at the 
national center to the moment the checked and verified data files and accompanying 
documentation were submitted to the IE A DPC. 

• The SITES 2006 School Coordinator Manual {lEAT)TC,2006c}descnhedth.ezctivities 
of the school coordinator — the person in the school responsible for organizing the 
SITES survey administration — from providing information about target population 
teachers to the time the completed materials were returned to the national center. 
This manual was subject to translation. 

• The SITES 2006 Data Management Manual (lEA DPC, 2006d) provided the NRC 
with instructions for entering and verifying the data as well as for the general use of 
software (see below). 

The lEA DPC supplied each NRC with three software packages to assist him or 
her with the data collection and to ensure uniform procedures: 

• The Within-school Sampling Software (WinW3S) served as a survey administration 
tool. WinW3S helped NRCs list and randomly sample teachers and target classes in 
each selected school, prepare forms to keep track of the sampled individuals, print 
labels for the personalization of questionnaires, and monitor the progress of the 
administration. The software stored all tracking information in a database so that this 
information could be used later to extract participation status, compute sampling 
weights, and verify the integrity of the sampling procedure. 

• The Data Entry Manager (WinDEM) enabled national center staffto adapt codebooks, 
capture and edit the SITES 2006 data through keyboard data entry, and perform 
a range of validity checks on the entered data prior to submission. The WinDEM 
database includes codebooks for each of the SITES 2006 questionnaires, providing 
all information necessary to produce data files for each instrument in a standard 
international format (see Chapter 9). 

• The lEA Survey System was used for converting the text pages in the paper questionnaires 
to online questionnaires (see Chapter 8) and delivering these to respondents. 

The installation and use of each software package is described in detail in the 
SITES 2006 Data Management Manual (lEA DPC, 2006d). In addition to receiving the 
manual, NRCs and their data managers received hands-on training in the use of these 
software packages from staff at the lEA DPC during a data management seminar held 
before the field trial in June 2005. 

7.4 Survey Forms and Identification Numbers 

Survey tracking forms were provided for tracking the participation of schools and, more 
importantly, for listing, sampling, and tracking teachers. The school coordinator s tasks 
included listing information about all mathematics and science teachers in the target 
grade, their possible exclusion from the survey, and information about all classes or 
courses taught in the target grade. The school coordinator was also required to estimate 
the percentage of teachers using ICT for their teaching. 
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To record the information, SITES 2006 utilized “teacher listing forms.” If teachers 
taught both mathematics and science to at least one class in the target grade, teachers 
were listed twice, once on the mathematics listing form and once on the science listing 
form. The information from these forms was entered into WinW3S at the national center, 
after which the random within-school teacher sample was drawn using the software. After 
sampling, national centers generated the “teacher tracking forms” on which all sampled 
mathematics and science teachers were listed. The teacher tracking forms were sent to 
schools, so that school coordinators knew to whom to distribute the instruments. 

It was essential that both tracking forms were completed accurately because 
they determined the eligible population and which questionnaire was given to which 
teacher. The tracking forms were used to facilitate data collection, record information 
essential to the computation of estimation weights, evaluate the quality of the sampling, 
and evaluate the quality of sampling within schools as well as the administration of the 
survey. All tracking forms were retained for review by staff at the lEA DEC. Appendix E 
of this report provides example forms. 

Based on the four-digit school IDs assigned by the lEA DEC sampling team, the 
WinW3S software created hierarchical identification numbers that uniquely identified 
principals (four digits), ICT coordinators (six digits), and teachers (eight digits). The 
teacher IDs clearly identified teachers for mathematics and for science. 

7.5 Administration of Questionnaires 

Each school principal was required to complete one “principal” questionnaire, and the 
ICT coordinator was assigned a “technical” questionnaire. Each teacher listed on the 
teacher tracking form was assigned a “teacher” questionnaire. Because there were no 
separate questionnaires for mathematics and science teachers, the labels attached to each 
questionnaire clearly identified both the subject domain (mathematics or science) as 
well as the reference/ target class context in which teachers were asked to complete the 
questionnaire (see Sections 4.3 and 6.7 for details). 

For each participating school, the NRC prepared a package containing all paper 
questionnaires and/or cover letters (see Chapter 8) for online administration, the 
teacher tracking form, and any other relevant materials prepared for briefing the school 
coordinators, for example, leaflets or letters of endorsement from ministries or unions. 
A set of labels and/or prepaid envelopes/parcels addressed to the national center was 
usually included to facilitate the return of the survey materials. 

School coordinators were then responsible for organizing the administration 
within the schools as follows: 

• Checking the materials when they arrived from the national center to ensure that 
there was one questionnaire corresponding to each teacher listed on the teacher 
tracking form; 

• Distributing the principal questionnaire for paper administration or the principal 
cover letter for online administration to the principal of the school; 

• Distributing the technical questionnaire for paper administration or the technical 
cover letter for online administration to the schools ICT coordinator; 

• Distributing the teacher questionnaires for paper administration or the teacher cover 
letters for online administration to the designated teachers as listed on the teacher 
listing forms; 

• Ensuring that the questionnaires were completed within the set administration 
period and reminding respondents to complete their questionnaires, if necessary; 

• Recording teacher participation information on the corresponding forms; 
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• Communicating to the national center any relevant information or obstacles ab out the 
administration in the school, especially any problems affecting teacher participation; 
and 

• Returning the completed and any unused instruments and the completed teacher 
tracking forms to the national center. 

7.6 Monitoring of Online Participation 

School coordinators were responsible for recording the return status of the paper 
questionnaires on the teacher tracking form. For the online questionnaires, school 
coordinators made an indication on the tracking form only if a teacher was assigned 
an online questionnaire. Tracking the completion status of all online questionnaires 
was done at the national center using the lEA SurveySystem Monitor component (see 
Chapter 8). The real-time status of all respondents who had already started filling in the 
questionnaire was monitored on a secure web page. If any teacher, school principal, or 
ICT coordinator expected to participate was not listed in the monitor, the NRC could 
ask school coordinators to follow up with the individual(s) concerned. 

After survey administration, national center staff copied the participation 
information from the lEA SurveySystem Monitor reports into WinW3S in order to 
record the participation status there. 

7.7 Material Receipt and Preparing for Data Entry 

In the period immediately following the administration of SITES 2006, the major NRC 
tasks included retrieving, collating, and verifying the integrity of the school materials. On 
receiving the survey materials from schools, the NRCs completed the following tasks: 

• Checked that the complete and appropriate questionnaires were received for every 
teacher listed on the teacher tracking form; 

• Verified that all identification numbers on all paper instruments were accurate and 
legible; 

• Cross-checked that the participation status recorded on the teacher tracking 
form matched the availability of questionnaires, the information on the paper 
questionnaires, and the information in the online monitor; and 

• Followed-up on schools that had not returned all the survey materials or for which 
forms were missing, incomplete, or otherwise inconsistent. 

At the national center, all necessary information about schools, ICT coordinators, 
and teachers were recorded in WinW3S. This information included the return status of 
the questionnaires. NRCs then organized the paper questionnaires and corresponding 
forms for data entry (see Chapter 9). 

7.8 Field Trial of Instruments and Procedures 

All of the above-mentioned procedures were field-trialed in the majority of participating 
education systems.' In spring 2005, this field trial survey was conducted to test the survey 
instruments, software, forms, and corresponding procedures. For this purpose, a sample 
of 25 schools was drawn in most education systems with the teacher-level sampling 
following the steps outlined in Chapter 6 of this report. 

As a result of the field trial, the instruments, the procedures, and the software were 
refined and improved where needed for the main study. In addition, valuable feedback 
was provided by individual NRCs, especially regarding strategies and best practices for 



1 Alberta (Canada); Ontario (Canada), and South Africa did not participate in the field trial. This was because the decision to participate 
in SITES 2006 was taken only after the field trial period ended. 
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achieving high response rates and clearing the survey project with regional authorities or 
gaining support from labor unions. 

The field trial also yielded results about the feasibility of the online data collection 
(ODC) in SITES 2006. A spilt-half design was used to compare data collected online 
and on paper and to guide decisions about the further use of the online and paper 
administration modes in a mixed-mode setting during the main study. Chapter 8 details 
the SITES 2006 ODC. 

7.9 Main Study Data Collection Periods 

As noted earlier in this report, administration of the SITES 2006 survey took place 
between March and June 2006 in the northern hemisphere and between September and 
October 2006 in the southern hemisphere. 

Because a number of education systems experienced substantial problems with 
securing high levels of school and teacher participation, mainly due to a situation of 
“survey fatigue” (lEA PIRLS 2006 and OECD PISA 2006 coincided with lEA SITES 
2006 and many national evaluation and assessments in many countries), a catch-up 
data collection period was granted in parallel to the southern hemisphere timeline in 
nine northern hemisphere education systems: Alberta (Canada), Catalonia (Spain), 
Denmark, Estonia, Finland, Hong Kong SAR, Israel, Norway, and Ontario (Canada). 
While the magnitude of the generally few additional cases did not allow for strong 
statistical investigation, explorations and reports concluded that no substantial response 
bias occurred because of the extension periods. 

7.10 Survey Activities Questionnaire 

After the data collection, NRCs were requested to complete the survey activities 
questionnaire to report their experiences during the administration and to collect 
important information about adherence to international procedures. This questionnaire 
was set up by the IE A DP C and administered online to NRCs immediately on completion 
of the data collection activities. The questions pertained to problems or unusual 
occurrences, if any, with respect to selecting the sample, securing school participation, 
translating or preparing the instruments, administering the questionnaires in the schools, 
and/ or creating and checking the data files. 

Responses to the survey activities questionnaire were carefully reviewed by 
the international consortium as a basis for data processing, cleaning, and analysis. 
The questionnaire responses did not yield major or critical problems and hence it was 
deemed unnecessary to take any special action with respect to the reported obstacles and 
the adjudication of data. 

Many NRCs confirmed that achieving high and acceptable response rates was a 
major challenge during the administration, mostly due to the nature of a survey in which 
respondents, in most cases, were able to withdraw from it on a voluntary basis. 
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8.1 Overview 

During planning for this third module of SITES, the research consortium decided that 
computer and internet technologies should he employed to assess ICT-related indicators 
and measures via two methodologically innovative components: (i) a computer- or weh- 
hased ICT skills assessment administered to students; and (ii) weh-hased questionnaires 
(Anderson & Plomp, 2001). Although the student component was eventually dropped 
due to operational obstacles, lEA confirmed its willingness to explore the feasibility 
of online data collection (ODC), stressing that the thematic background of SITES 
2006 made the study a good candidate for employing survey technology similar to the 
technologies and tools used in pedagogical contexts under focus in SITES. 

The consortium expected that Grade 8 mathematics and science teachers would 
be receptive to using ICT as a more convenient, interesting, or simply “up-to-date” mode 
of survey administration. The consortium also expected that the online mode would offer 
operational benefits, lower the individual and overall response burden, and significantly 
reduce paper handling and data-entry costs for national centers. Moreover, the online 
mode was expected to yield a more accurate and reliable international database. Finally, 
ODC was seen not only as a means of conducting and supporting research, but also as 
an object of research itself, in terms of addressing the question of what constitutes the 
characteristics of a working ODC system, as used in an international comparative survey 
(compare, for example, Couper, 2000, and Dillman & Bowker, 2001). 

Several of the countries participating in SITES 2006 had used ODC; for most of 
them, the administration of electronic questionnaires to schools and teachers had become 
commonplace and routine. However, until the time of this study, large-scale educational 
surveys at the international level were based entirely on paper questionnaires. If this new 
experience were to be successful, it had to meet the standards that lEA had established 
for its studies (Martin, Rust, & Adams, 1999). This chapter hence provides information 
on the conceptual considerations, the procedures, and the resulting staged technical 
implementation. A second important concern presented in this chapter centers on the 
question of whether the different data- collection modes would introduce effects and 
biases into the data that could influence (adversely or otherwise) meaningful statistical 
analyses. 

8.2 Conceptualization and Mixed-mode Considerations 

To correctly sequence work steps and to ensure comparability of data, paper versions 
of the three questionnaire types used in SITES 2006 had first to be finalized in terms 
of translation and layout verification, even if the expectation was that all or nearly all of 
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the data would be collected online. From these final paper versions, the questionnaires 
were converted for the online mode followed by final optical and textual verification. The 
electronic versions of the SITES questionnaires could only be filled in via the internet. 
No other options were allowed, such as sending pdf documents via email or printing out 
the online questionnaires and mailing them to the national center. 

In addition to procedural considerations, the design had to address certain 
technical issues. Respondents needed only an internet connection and a standard internet 
browser. No additional software was required to fill in the questionnaire (see “Technical 
Implementation” below). Because the focus of SITES 2006 was on pedagogical use of 
ICT and not on computer literacy, the terminology used and technical hurdles were 
carefully considered and implemented in a way that reduced, to the very minimum, the 
computer skills respondents needed to access and answer the questions. 

Because SITES 2006 was conducted using a mixed-mode design, data from 
different collection modes had to be merged to a single set within and across systems. 
Consequently, potential sources of error originating from the use of the two parallel 
modes had to be controlled for and reduced as much as possible to ensure uniform 
and comparable conditions across modes as well as countries. The design established 
several general similarities, and questionnaires in both modes were self- administered 
and equally situated in the visual domain, in contrast to mixed-mode surveys that 
simultaneously employ self-administered questionnaires and telephone or face-to-face 
interviews. Moreover, respondents were identified by the same sample design, contact to 
respondents was established by similar means, and data from both modes were collected 
over the same period of time. Great care was taken to present questions in ways that are 
easy to read on screen and self-explanatory to complete. 

The navigational paradigm for the online questionnaire was designed to be as 
similar as possible to that of the paper questionnaires. Respondents could use “next” and 
“previous” buttons to navigate to an adjacent page, similar to flipping physical pages. In 
addition, the implementation of a “table of contents” mirrored the capability of opening 
a specific page or question of a paper questionnaire. While most respondents followed 
the sequence of questions direcdy, the two features allowed respondents to skip or omit 
questions just as they would have if answering a self-administered paper questionnaire. 
To further ensure the similarity of the two sets of instrumentation, responses to the 
online questionnaires were neither made mandatory nor evaluated in detail (e.g., using 
hard validations). 

However, certain differences in the representation of the two modes remained. 
Rather than presenting multiple questions per page, the online questionnaire was 
presented question by question because of the complexity associated with the large 
number of questions. While the visual or sensory impression of the length and burden 
of a paper questionnaire can be estimated easily, the online questionnaires attempted to 
offer this through a “table of contents” that listed each individual question and progress 
counters. Multiple-choice questions were implemented with standard HTML “radio 
buttons.” While it was possible for respondents to change the answer to any other 
option, it was not possible for them to uncheck the answer completely as they could in 
the paper questionnaires by crossing out (cancelling) a given answer. The consortium 
acknowledged the possibility to add extra “don’t know” or “cancel” categories to all 
such questions, but took a balanced decision against it because the level of “cancelled” 
responses typically observed, including in the SITES 2006 field trial, was extremely low 
or negligible. 

Overall, a near-identical representation between modes (Denscombe, 2006) was 
achieved, an accomplishment that enabled the yielding of identically structured and 
comparable data. 
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8.3 Technical Implementation 

After addressing the procedural requirements and methodological necessities and 
constraints, the consortium determined a plan for implementation. No single “off- 
the-shelf” solution could he found that would satisfy all survey operations and 
requirements, most importantly in the areas of (i) decentralized translation, adaptation, 
and documentation, (ii) mixed-mode data collection and subsequent data processing, 
and (iii) minimal prerequisites on the side of respondents’ or schools’ computers. In mid 
2004, the consortium accordingly delegated to the lEA DPC and its software unit the 
task of developing a suitable software (later coined the “lEA SurveySystem”). 

The lEA SurveySystem is a hierarchical model of a survey that stores and 
manages all questionnaire-related information, including text passages, translations, and 
adaptations, verification rules, variable names, and information for data management 
(see Figure 8.1). The SurveySystem’s consolidation of metadata in a single set of files that 
the SITES 2006 national and international centers could easily send to one another over 
the internet allowed for a consistent way of managing the localized online versions of the 
questionnaires. 



Figure 8.1: Architectural Overview of the SurveySystem 




To serve the different usage scenarios, three distinct components of the 
SurveySystem were developed. The Designer was used to create, delete, disable, and edit 
survey components (e.g., questions and categories) and their properties. It allowed for 
translation of all text passages in the existing national paper questionnaires and additional 
system texts, and it included a complete web server to verify and test-drive the survey 
exactly as if under live conditions. The Designer also supported the export of codebooks 
to lEA’s generic data-entry software WinDEM to allow for isomorphic data entry of 
online and paper questionnaires. 

The Web component was a compiled application that served questionnaires in 
HTML format to the respondents for completion from within standard internet browsers. 
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Given the overall goal of securing the maximal possible coverage, no respondents were 
to he excluded because of incompatible browsers or disabled features. Because computer 
literacy was likely to vary greatly among groups of respondents, the design sought a 
balance between minimally desirable capabilities and simplicity. This achievement was 
especially important, as requirements in terms of connection speed, available software 
(browsers), and modem and processing speed were identified as crucial obstacles during 
initial discussions and the review of literature (see, for example, Reips, 2002). In this 
sense, the approach taken in SITES 2006 (selected aspects below) was similar to that of 
the “respondent-friendly design” explicated by Dillman, Tortora, and Bowker ( 1998). 

In detail, the output was tested to assure (near) identical representation at 
minimum screen sizes in all supported browsers, which minimally were required to 
support HTML 4.0 (a standard since 1998), the “dir” attribute for bi-directional Unicode 
text, and cascading style sheets for basic formatting. With the exception of the welcome 
screen, graphics were not used. The extremely few users with browsers that did not meet 
this requirement received a translated list of supported browsers and information on 
contacting their national center. The Web component made use of plain HTML controls 
only and therefore did not require “fancy” technologies such as cookies, JavaScript, 
Flash, and/ or pop-ups. 

Finally, the web-based Monitor component allowed national centers to audit 
participation in real-time. It also allowed the centers to follow up schools in the case of 
incomplete or not-returned questionnaires in a similar way to that used for administration 
of the paper questionnaires. 

All systems were programmed on the basis of Microsoft’s .NET framework 
because of its robustness and excellent support for multilingual (Unicode) and internet 
applications in general. The live systems were hosted on dedicated high-performance 
servers rented from a reliable and experienced solution provider in Germany. Appropriate 
measures were taken to secure the data, and these were further strengthened by a 
professional security audit conducted by an external organization to verify the final 
setup. The lEA DPG developed backup and disaster recovery strategies and constantly 
monitored the systems for permanent availability during the data- collection period. 

8.4 Operations 

After they had successfully passed the translation and layout verifications (see Chapter 
5 ), the national centers used the SurveySystem Designer software to convert the 
questionnaires for the online mode. In addition to being sent the software, the national 
centers received the international English survey files prepared by the consortium. The 
lEA DPG provided detailed documentation as well as corresponding training as part of 
a data management seminar for this task. The conversion to the online mode was based 
on the concept of “cultures,” a certain language within a certain cultural context. For 
example, instrumentation for “Estonian in Estonia” and “Russian in Estonia” had to be 
prepared separately. Because the translation was already verified and fixed for the paper 
questionnaires, this conversion was mainly a copy-and-paste procedure. In addition to the 
questionnaire passages, certain translations were needed exclusively for online purposes, 
such as texts on the welcome screen and on navigation buttons or for error messages. 
Before submitting the files to the lEADPC, the national centers were required to perform 
an optical side-by-side comparison using the integrated preview component. 

After receiving from the national centers the files containing all structural 
information, translations, and national adaptations needed to run the online survey, 
staff at the lEA DPG performed a comprehensive optical question-by-question check 
for differences between the online and paper versions as an additional quality control 
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measure prior to uploading a country’s survey Any detected deviations, such as mistakes 
in copying passages into the correct location, were reported hack to the national centers. 
The online questionnaires were approved and made accessible only after any remaining 
issues had heen resolved satisfactorily, thereby ensuring an isomorphic representation of 
questions in both modes. 

To limit the administrative burden and required school communication, the 
decision on whether to assign the online or the paper questionnaire as a default to 
respondents was initially taken at the national center and was based on prior experience 
gained from participation in similar surveys and the SITES field trial. In most of the 
systems, the default mode was set at the school level. Every respondent of any such 
school — the principal, the ICT coordinator, and the sampled teachers — were assigned 
to the same mode, online or paper. In other countries, the default mode was alternatively 
set at the questionnaire level, so that all respondents of the same group (principals, ICT 
coordinators, and teachers) received the questionnaire in the same mode. However, the 
NRCs were required to take into account the mode that a specific school or individual 
preferred. The NRCs also had to ensure that every respondent assigned to the online 
mode by default had the option to request and complete a paper questionnaire regardless 
of the reasons for not being willing or able to answer online. 

To ensure confidentiality, every respondent received individual login information. 
The national centers sent this information, along with general information on how to 
access the online questionnaire, to respondents in the form of “cover letters.” In line 
with the procedures for the paper questionnaires, the information was distributed to 
the designated individuals via the school coordinator. During the administration period, 
respondents could log in and out as many times as needed and resume answering the 
questionnaire at the question they had last responded to in their previous session. 
Answers were automatically saved whenever respondents moved to another question, and 
respondents could change any answer at any time before completing the questionnaire. 
During administration, support was given by the national center, which, in turn, could 
contact the lEA DEC if unable to solve the problem locally. 

National centers were able to monitor the responses to the online questionnaires 
in real-time and to send reminders to those schools where people had not responded in 
the expected period of time. School coordinators could then be asked to follow up with 
the individuals concerned. 

Although education systems using the online mode in SITES 2006 faced parallel 
workload and complexity before and during the data collection, they had the benefit 
of a reduction in workload afterwards. Because answers to online questionnaires were 
already in electronic format, and responses were stored on servers maintained by the lEA 
DPC, there was no need for separate data entry. 

In summary, the procedures that were necessary to support ODC alongside 
the conventional paper-and-pencil track were designed in such a way that little or no 
changes were required with respect to the tried-and-true survey operations typically 
employed in lEA surveys. The main challenges were to cater for isomorphic versions 
of the instrumentation in both modes, to reliably administer the resulting mixed-mode 
survey, and to subsequently integrate the two data sources. The overall conclusion is that 
SITES 2006 was successful in achieving this. 

8.5 Staged Development, Implementation, and Evaluation 

The introduction of ODC technology and methodology in SITES 2006 took place in a 
gradual and careful way. The ODC component was launched in three main phases — a 
technical try-out, the field trial, and the main survey. 
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The first phase ( 1 ), the technical try-out with 1 2 participants in May 2005, sought 
to collect basic experience and to locate and eliminate basic technical problems with 
respect to languages, the conversion of instruments from paper to online, scripts and 
script direction (e.g., Hebrew), and browser compatibility In general, this phase was 
highly successful in identifying key requirements and defining further development. 

The second phase (2), the field trial in autumn 2005 with 16 out of the 18 
participants, made use of a feature- complete ODC system developed on the basis of the 
technical try-out findings. Prior to the field trial, NRCs were asked whether they intended 
to use ODC for the main data collection. If they did, these systems were obliged to field- 
trial the ODC procedures as well. 

The largest part of the evaluation was based on the findings and outcomes of 
the field trial, especially in terms of any new procedures developed. While producing 
a working technical and procedural solution was, of course, a key prerequisite, the 
evaluation of the online approach in SITES 2006 was based on existing evidence (where 
applicable and appropriate in general), but required separate investigation to reflect the 
unique SITES 2006 characteristics by addressing two key questions: 

• Key Question 1 — feasibility: Investigate whether ODC methodology and procedures 
can work in the context of an international lEA study. 

• Key Question 2 — validity: Analyze whether the two modes (online and paper and 
pencil) yield comparable data, thus allowing the implementation of both modes in 
and across countries. 

The following list contains selected elements of the evaluation of the feasibility: 

• There were no observed or reported problems with respect to translation, conversion, 
and representation of advanced scripts (such as Thai, Chinese, and Hebrew) or script- 
direction (right-to-left writing) during preparation and administration of either the 
field trial or the main study. 

• On average, conversion and verification required two to three person days per 
language version, equivalent to one day per questionnaire, with additional time 
needed to communicate and resolve the few and far between differences identified 
during the thorough side-by-side comparison of instruments at the lEA DEC. All 
such problems were resolved successfully before administration. 

• During administration, no major problematic behavior of servers (e.g., unplanned 
down-time or hacking) or users’ browsers was observed or reported for the 
approximately 26,000 questionnaires in the main study. 

• For the field trial, the review of server log files showed that, in relation to all online- 
administered questionnaires, no more than 0. 1 % of users attempted to access the web 
application with outdated browsers; a similar magnitude was observed during the 
main study. 

• The SITES 2006 procedures required paper questionnaires to be used in case 
respondents refused or simply were not in a position to participate online, for example, 
because of lacking infrastructure. Unfortunately, during the field trial, the distribution 
of these fail-back paper questionnaires seemed not to have worked for some systems. 
The consortium regarded the need to reliably record and manage the demand for 
paper instead of online questionnaires as highly important. The procedures were 
discussed with the NRCs, who firmly monitored the successful application during 
the main data collection. 

• National centers reported making extensive use of the monitor application to facilitate 
follow up of non-respondents. 

• Eventually, the integration and subsequent cleaning and processing strategies were 
adjusted to reflect the online data source, but did not constitute an obstacle in respect 
to quality. In all cases, manually entered datasets and data from the online mode 
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Structurally matched 100%. That is, within any one participating system, both modes 
used the exact same variable and coding scheme. 

• There were no indications or reports that reluctance or even resistance on the part 
of schools to participate in the survey or the subsequent willingness of individual 
respondents to fill in the questionnaire related to the mode of administration itself, 
in this case “online.” In some of the participating systems, difficulties in securing 
acceptable school participation rates were instead due to a general atmosphere of 
“survey” or “educational reform fatigue.” 

Overall, the majority of procedures and tasks, supported by manuals and direct 
support via email, went as intended, even though, for some areas, more attention and 
allocation of time and resources were required than originally expected, for instance, for 
cross-checking questionnaire wording and structure. 

Evaluating whether the paper and online modes yielded comparable data was a 
more complex process (Brecko & Carstens, 2007). For the purpose of comparison, a 
split-sample design at school level was implemented during the field trial (with a typical 
sample size of 25 schools combined within each system). Statistical analyses were 
conducted and included — but were not limited to — the investigation of response rates, 
drop-out, indicator reliability, and the mode independence of nominal as well as interval 
measures. 

There were no evident substantial differences between the data derived from 
paper and the data derived from the online mode that would reduce the ability to merge 
these sets of data and to make joint analyses. Issues were identified in terms of drop-out, 
which were partially rooted in the preliminary character of the field trial instruments. 
The level of drop-out was negligible or fairly limited in the two shorter school-level 
questionnaires. The level was substantial, although not critical, in the significantly longer 
teacher questionnaire. Adequate measures, in this case the reduction of overall length, 
were implemented for the main study. 

The consortium hence decided to recommend the further use of ODC during 
the main study where possible and desired. Accordingly, in this third and final phase in 
April/May 2006 (northern hemisphere) and August/September 2006 (southern 
hemisphere), ODC was offered as an international option, making SITES 2006 the 
first lEA study — and the first in the history of international comparative educational 
assessments — to make ODC an integral component of its framework and operations. 

8.6 Main Study Participation and Mode Distribution 

While nearly all of the participating systems expressed at least some interest in using the 
online mode as eitherthe default for data collection (withpaper-and-pencil questionnaires 
as a mandatory fail-back) or for only selected schools, regions, or individuals, 17 of the 
22 participating systems implemented ODC, usually as the default mode for collecting 
data, after taking into account its suitability for their local context. The consortium 
recommended that the decision accord with the level of confidence that each system had 
in regard to using the online mode, based on factors such as the within-country computer 
and internet penetration, poor online response rates in previous surveys, and (most 
importantly) the outcomes of the mandatory field trial. One participant was advised to 
refrain from using ODC given unfavorable experiences in previous similar designs in 
the past and in the field trial. The remaining four participants (Chinese Taipei, Japan, 
Ontario (Canada), and South Africa) decided not to use ODC because of feasibility 
concerns. Estonia and Finland administered the survey in two languages each, resulting 
in a total of 19 sets of instrumentation. 

Table 8.1 provides the distribution of paper and online questionnaires in all 
participating education systems. The distributions for principals, ICT coordinators, and 
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science teachers were highly similar within each system. It is noteworthy that a number 
of systems managed to collect all data entirely using online questionnaires yet did not 
compromise coverage. 



Table 8.1: Extentto whichPaper and Online AdministrationModesWerellsedfor Mathematics 
Teacher Questionnaires during the Main Study 



Education System 


Paper 


Online 


Chinese Taipei 


100.0% 


0.0% 


Japan 


100.0% 


0.0% 


Ontario, Canada 


100.0% 


0.0% 


South Africa 


100.0% 


0.0% 


France 


78.9% 


21.1% 


Russian Federation® 


58.9% 


41.1% 


Denmark 


30.0% 


70.0% 


Slovenia 


26.6% 


734% 


Chile 


12.4% 


87.6% 


Moscow, Russian Federation® 


10.1% 


89.9% 


Slovak Republic 


7.1% 


92.9% 


Hong Kong SAR 


6.2% 


93.8% 


Catalonia, Spain 


2.4% 


97.6% 


Italy 


1.2% 


98.8% 


Finland 


0.9% 


99.1% 


Alberta, Canada 


0.0% 


100.0% 


Estonia 


0.0% 


100.0% 


Israel 


0.0% 


100.0% 


Lithuania 


0.0% 


100.0% 


Norway 


0.0% 


100.0% 


Singapore 


0.0% 


100.0% 


Thailand 


0.0% 


100.0% 


Average 


28.9% 


71.1% 



Note: 

* The table reflects the questionnaire modes as indicated in the international database. Certain schools from the Moscow 
and Russian Federation samples contributed to one another’s estimates (see Chapter 10). At the time of data collection, 
however, all data for the Moscow region were collected online and all data for the Russian Federation were collected via 
paper questionnaires. 



Table 8.2 provides an overview of the mode distribution for completed 
questionnaires in (i) all education systems^ and (ii) in those systems that used ODC 
during the main data collection. As can be seen, the proportion of paper versus online 
mode was highly consistent across questionnaire types, for example, for principals and 
ICT coordinators. The 17 countries that opted to use ODC actually administered the 
vast majority of the questionnaires (about 88%) in that mode. Overall (i.e., in all 22 
systems), about 72% of the international data available to SITES 2006 originated from 
online questionnaires. 
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Table 8.2: Extent to which Online and Paper Questionnaires Were Used by all Participating 
Education Systems (left) and by Systems Opting for ODC (right) 
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9.1 Overview 

Creating the SITES 2006 international database (IDB) and ensuring its integrity required 
close coordination between and cooperation among the project and sampling staff at the 
lEA Data Processing and Research Center (DPC) and the SITES 2006 international 
coordinating center (ICC), as well as with the national research coordinators (NRCs). 
The primary goals were to ensure that all national information in the international 
database conformed to the international data structure and coding scheme, that any 
national adaptations to questionnaires were reflected appropriately in the codebooks 
and the corresponding documentation, and that all variables used for international 
comparisons were indeed comparable across all education systems. Quality control 
measures were applied throughout the process. 

This chapter describes the data-entry and verification tasks undertaken by the 
national centers, the integration of data from the paper and online administration modes, 
the data-editing and database-creation procedures implemented by the lEA DPC in 
collaboration with the international consortium, and the steps taken at all involved 
centers to confirm the integrity of the international database. 

9.2 Data Entry and Verification at National Research Centers 

Each SITES 2006 national research center was responsible for transcribing the 
information from the three types of questionnaires (principal, technical, and teacher) 
administered to the samples of schools and teachers into computer data files. The 
lEA DPC supplied national centers with the Windows Data Entry Manager software 
(WinDEM) and corresponding documentation in the SITES 2006 Data Management 
Manual (lEA DPC, 2006). In addition, the lEA DPC held a four-day data management 
seminar in Hamburg, Germany, in June 2005, on software, procedures for national 
adaptations, and rules and procedures for data entry. The seminar was specifically 
targeted at the person within the national team responsible for data management and 
liaising with the lEA DPC. 

The responses from the principal, technical, and teacher questionnaires were 
entered into data files created from internationally predefined codebooks, which 
contained information about the names, the lengths, the locations, the labels, the valid 
ranges (for interval/ratio measures) or valid values (for nominal/ordinal questions), and 
the missing codes for each variable in each of the three questionnaire types. Before data 
entry could commence, data managers were required to adapt the codebook structure 
to reflect any adaptations made to the national questionnaire versions, for example, 
a nationally added category. These adapted codebooks then served as templates for 
creating the corresponding data-entry file(s). 
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In general, the national centers were instructed to discard any unused or empty 
questionnaires, and to enter any questionnaire that contained at least one (l) valid 
response. To ensure consistency across education systems, the basic rule for data entry 
in WinDEM required national staff to enter data “as is” without any interpretation, 
correction, truncation, or cleaning. The resolution of any inconsistencies remaining after 
the stage of data entry was delayed until the stage of data “cleaning” (see helow). 

The general rules for data entry were as follows: 

• Responses to categorical questions were generally coded as “1” if the first option 
(checkbox) was used, “2” if the second option was marked, and so on. 

• Responses to “check-all-that-apply” questions were coded as either “1” (checked) or 
“2” (not checked). 

• Responses to numerical or scale questions (e.g., school enrolment) were entered also 
“as is,” that is, without any correction or truncation, even if the value was outside of 
the originally expected range, for example, if an ICT coordinator reported that he or 
she spent 80 hours a week on ICT-related support. 

• Likewise, responses to filter questions and filter-dependent questions were entered 
exactly as filled in by the respondent, even when the information provided — however 
obvious — was logically inconsistent. 

• If responses were not given at all, were not given in the expected format, were 
ambiguous, or were in any other way conflicting (e.g., two options in a multiple- 
choice question were selected), the corresponding variable was coded as “omitted or 
invalid.” 

• Unlike other lEA surveys concerned with measures of student achievement, SITES 
2006 did not use a code to identify “not administered” questions, for example, those 
that were misprinted. Also, in these infrequent cases, the “omitted or invalid” code 
was used. 

In a parallel step to that for the questionnaire data entry, the data manager at each 
national center used the information from the teacher tracking forms (see Chapter 6) to 
verify the completeness of the materials. Participation information, for example, whether 
the concerned teacher left the school permanently between the time of sampling and the 
time of the actual administration, was entered in the WinW3S within-school sampling 
software (see Chapter 6). 

In order to check the reliability of the data entry within the education system, 
national centers were required to have at least 40 completed principal questionnaires, at 
least 40 technical questionnaires (school level), and at least 100 teacher questionnaires 
entered twice by different staff members as early as possible during the data capture 
period. This procedure allowed data managers and the lEA DPC to identify possible 
misunderstandings or mishandlings of data-entry rules and to initiate appropriate 
reactions, for example, the re-training of staff. The acceptable level of disagreement 
between the originally entered and the double-entered data was established at 1% or 
lower. The margin of error observed for all education systems participating in the SITES 
2006 main data collection was well below this threshold. 

Before sending the data to the lEA DPC for further processing, national centers 
were responsible for carrying out mandatory verification steps on all entered data and 
for undertaking corrections as necessary. The corresponding routines were included 
in the WinDEM program, mainly in order to identify invalid data, but also to check 
the consistency among records. For example, data files were checked for duplicate 
identification codes or data outside the expected valid range or the values defined as 
valid. Data managers were required to review the corresponding reports and to resolve 
any inconsistencies and, where possible, correct problems by looking up the original 
survey questionnaires. 
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While the lEA DPC strongly encouraged every education system to use the 
WinDEM software for data entry to meet all standards and rules, a few participating 
education systems used a different data-entry system, such as one routinely used hy an 
external survey company. However, these systems were nonetheless required to conform 
to all specifications established in the international codehooks and to verify their data 
using the same consistency checks as defined within the WinDEM software. Data 
entry in France and Japan was completed outside of WinDEM, hut submitted in the 
internationally required format together with evidence of the accuracy and reliability of 
the data entry as it would have been if produced by the international procedures. 

In addition to the data files described above, national centers were requested 
to provide detailed data documentation to the lEA DPC. This material included hard 
copies or electronic scans of all original teacher tracking forms, electronic copies of the 
national versions of all questionnaires, the final national adaptation forms (NAFs), and 
a report on data-capture activities collected as part of the survey activities questionnaire 
(SAQ). 

9.3 Data Checking, Editing, and Quality Control at the lEA DPC 

Once the data were submitted to the lEA DPC, a process generally referred to as “data 
cleaning” commenced. The main objective of the process was to ensure that the data 
adhered to international formats, that information from principals, ICT coordinators, 
and teachers could be linked across different survey files, and that the data accurately 
and consistently reflected the information collected within each education system. 
The lEA DPC went to great lengths to ensure that the data received from the SITES 
2006 participants were of high quality and internationally comparable. The foundation 
for quality assurance was laid before the data first arrived at the lEA DPC through the 
provision of software designed to standardize a range of operational and data-related 
tasks. 

• The WinW3S software performed the within-school sampling operations, adhering 
strictly to the sampling rules defined by SITES 2006. The software also created 
all necessary listing/tracking forms and stored any school- or teacher- specific 
information, such as gender and participation status. 

• The WinDEM software enabled entry of all questionnaire data in a standard, 
internationally defined format. The software also included a range of checks for data 
verification. 

A complex study such as SITES 2006 requires a correspondingly complex 
data-cleaning design. To ensure that programs ran in the correct sequence, that no 
special requirements were overlooked, and that the cleaning process was implemented 
independently of the persons in charge, the following steps were undertaken. 

• Before being used with real data, all data-cleaning programs were thoroughly tested 
using simulated data sets containing, as much as possible, expected problems and 
inconsistencies. 

• All incoming data and documents were registered in a database. The date of arrival 
was recorded, along with any specific issues meriting attention. 

• All national adaptations and all detected deviations from the international data 
structure were recorded in a “national adaptation database” and verified against the 
structure and content of the data itself (The reports from this process are available 
for data analysts in Appendix D.) 

• The cleaning was organized according to strict rules applied to all national data sets so 
that deviations in the cleaning sequence were impossible. 

• All systematic or manual corrections made to data files were implemented in SAS and 
recorded in specific cleaning reports for consortium and NRC review. 
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• Once the data cleaning was completed for an education system, all cleaning checks 
were repeated from the beginning to detect any problems that might have been 
inadvertently introduced during the cleaning process itself 

9.5.7 Import, Documentation, and Structure Check 

Data cleaning began with an analysis of the submitted data-file structures and a review 
of data documentation consisting of the teacher tracking forms. Most education systems 
submitted all required documentation along with their data, which greatly facilitated the 
data checking. The lEA DPC contacted those systems that returned incomplete data 
and/ or documentation. 

Next, all available codebooks and data were imported from the source files and 
combined into SAS databases. Again, each questionnaire type corresponded to one 
SAS database and one SAS codebook file. In this step, both the data originating from 
paper questionnaires as well as online questionnaires were combined and checked for 
structural agreement (see Chapter 8 on online data collection). The data from both 
administration modes were structurally equivalent and also made use of the same valid 
and missing codes in all cases. The early combination of these data in the import stage 
ensured that data resulting from both administration modes were fed through the same 
data-processing systems and checks as described in the remainder of this chapter. 

The first checks implemented at the lEA DPC looked for differences between 
the international file structure and the national file structures. As described above, some 
countries made structural adaptations to the questionnaires; the extent and nature of 
such changes differed greatly across education systems. While some systems administered 
the questionnaires without any changes, except for translations and necessary cultural 
adaptations, others inserted questions or options within existing international variables 
or added entirely new national variables. Given the associated risk of deviating from 
the international data structure, NRCs wishing to make such changes were required to 
follow certain strict rules to allow unequivocal integration of nationally adapted variables 
for international comparison. 

In general, the extent of adaptations made to the international questionnaires 
was fairly low compared to other lEA surveys. Where necessary, the lEA DPC modified 
the record layout and/ or values to ensure that the resulting data were internationally 
comparable. For instance, additional national options in multiple- choice questions were 
recoded in such a way that they adhered to the international code scheme. 

The NRCs and data managers received detailed reports on any identified 
structural deviation together with documentation on how the lEA DPC resolved the 
deviations. In the case of national adaptations, data were recoded back to the required 
international values, and national variables were created to hold the original values for 
later use in national reports. In a few cases, data were not available for certain variables 
because the corresponding question was not administered nationally (see Appendix 
D). In most of these cases, usually in the case of “tracked” school systems (Question 
3 of the teacher questionnaire), data were systematically recoded to a particular value 
agreed with the NRC. There was only one case in which data had to be removed from the 
international database, and this was because the information was no longer internationally 
comparable. 

9.5.2 Identification Variable and Linkage Cleaning 

To identify, track, and document each participant and each corresponding questionnaire 
in a survey, each record in a data file needs to have a unique identification number. The 
existence of records with duplicate ID numbers in a file implies an error of some kind. 
In SITES 2006 , if two records shared the same ID number, and contained exactly the 
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same data, one of the records was deleted and the other remained in the database. If the 
records contained different data (apart from the ID numbers) and it was impossible to 
identify which record contained the “authentic” data, and if consultations with the NRC 
did not resolve the matter, both records were removed from the database. The lEA DPC 
deleted data in a very small number of cases only. In addition, only a tiny number of 
records present in both the paper and the online data files were identified. 

In SITES 2006, data collected at the school level were recorded in two files — the 
principal file and the technical file. It was crucial that the records from these files could 
be linked together correctly, that is, 1 : 1. In addition, the school-level data were linked to 
the multiple teacher-level records for that school, that is, Im. In both cases, the linkage 
was implemented through a hierarchical ID numbering system and was cross-checked 
against the tracking forms and corrected where necessary. A special requirement in 
SITES 2006 was to identify teachers teaching science as well as mathematics. These 
teachers were consequently listed and, in a few cases, sampled for both populations. 
Secondary identification variables had earlier been used to link the records for both 
questionnaires from the same data-entry file. Therefore, if only one of the two assigned 
questionnaires was returned, for instance, in the mathematics context, all person-related 
variables, independent of the subject or target class context, such as age or gender, were 
copied to the corresponding empty record, in this example to the science teacher file, to 
allow for more precise population estimates during the analysis stage. 

Further ID cleaning focused on consistent tracking of information between the 
data used for listing, sampling, and tracking in WinW3S and the actual responses in the 
questionnaire. Where necessary, variables pertaining to the teachers’ gender, year of 
birth, exclusion status, and participation status were verified and checked against the 
original paper teacher tracking forms. 

Where applicable and possible, close cooperation with the national center was 
sought to resolve any ID or linkage inconsistencies. For this purpose, NRCs and data 
managers received standardized reports comprising each identified inconsistency. 
Once the ID, linkage, participation, and exclusion information was finalized, data were 
transferred to the lEA DPC sampling unit and used to calculate participation rates, 
exclusion rates, and, finally, estimation sampling weights. 

9.3.3 Resolving Inconsistencies in Questionnaire Data 

After each data file was matched to the international standard, as specified in the 
international codebooks, a series of standard cleaning rules were applied to the files. This 
process was conducted through use of software (developed at the lEA DPC) that can 
identify and, in many cases, automatically correct, inconsistencies in data. Details about 
all implemented cleaning checks and procedures and any actions applied to the data were 
given to the national centers as part of a comprehensive data-processing documentation 
and were explained during the third NRC meeting in April 2007. 

Filter questions, which appear in certain positions in the questionnaires, were 
used to direct the respondent to a particular question or section of the questionnaire. 
Filter questions and their dependent questions were treated automatically in most 
cases. If the filter question contained a value and the dependent questions were validly 
skipped, dependent variables were coded as “logically not applicable.” If a response to a 
filter question was either omitted or equivalent to “no,” thereby marking the dependent 
questions as not applicable, and yet the dependent questions were answered in an 
unambiguous pattern, the filter question was recoded to the equivalent of “yes” or 
“applicable.” 

Split variable checks were applied to questions where the responses were coded 
into several variables. For example. Question 26 of the principal questionnaire listed a 
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number of developments and asked principals to mark whether they played a particular 
role in them by checking “yes.” Occasionally principals marked the “yes” boxes but 
left the “no” boxes unchecked, resulting in “omitted” values in the data file. Because, 
in these cases, it could be assumed that the unmarked boxes actually meant “no,” the 
corresponding variables were imputed accordingly. Similar rules were applied to partially 
answered “check-all-that-apply” questions, for example. Question 12 from the technical 
questionnaire, or partially answered numerical questions, for example Question 7 of the 
technical questionnaire. In some cases, the SITES 2006 questionnaires called for even 
more complex recoding rules, for instance, in so-called multi-matrix questions with yes/ 
no lists that were also used as filter questions (e.g.. Question 15, teacher questionnaire). 

Finally, variables within and across data files were verified against one another to 
identify and resolve inconsistent response patterns or multivariate oudiers. For example. 
Question 5(a) in the technical questionnaire asked for the total number of computers 
available in the school, while 5(e) asked for the number of computers connected to the 
internet. Clearly, the number given for 5(e) should not have exceeded the number given 
for 5(a). Question 1 of the teacher questionnaire (class enrolment) was another example 
in which the IE A DPC attempted to identify implausibly high values. 

The number of inconsistent and implausible responses in the data files varied from 
system to system, but no national data were completely free of inconsistent responses. 
Each problem was recorded in a database, identified by a unique problem number along 
with a description of the problem and the action taken by the program or by the staff of 
the lEA DPC. Issues that could not be corrected using systematic rules were reported 
back to the NRC so that original data-collection instruments and tracking forms could 
be checked to trace the source of the inconsistency. Wherever possible, staff at the 
lEA DPC suggested a solution and asked the NRCs either to accept it or to propose 
an alternative. Data files then were updated to reflect the solutions agreed on. Both 
systematic corrections as well as those apparent on a case-by-case level were applied 
directly in SAS program syntax and carried out automatically for each cleaning run. 

Where the NRC could not solve problems by inspecting the instruments and 
forms or could not suggest a satisfying solution or explanation, final cleaning rules were 
defined by the ICC, usually by recoding data (e.g., computer counts) using auxiliary 
information. In some instances in which a clear and unambiguous decision was not 
possible, for instance with respect to the number of students in a class, the data remained 
in the files unchanged, reflecting previous lEA practices. Users of the international 
database were cautioned in the user guide to review any concerned variable prior to 
analysis and to exclude cases deemed to be inappropriate or implausible. 

9.3.4 Handling of Missing Data 

During the SITES 2006 data entry using WinDEM at the national centers, two types 
of entries were possible: valid data values and missing data values. Data-entry staff were 
able to assign either the valid values or a value for “omitted/invalid.” Later, at the lEA 
DPC, additional missing values were applied to the data to be used for further analyses 
and to differentiate different response behaviors. 

In the international database, five missing codes were used: 

• Omitted/ invalid (9): The respondent had a chance to respond to the question, but did 
not do so or provided an invalid response. The value was also assigned in extremely 
rare cases where questions were misprinted or otherwise not legible. 

• Not administered (8): The respondent was not administered the actual question, item, 
or option because it was removed from the national version. He or she had no chance 
to read and answer the question. 
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• Not reached (7): This code indicated variables not reached by the respondents at 
the end of a questionnaire, usually due to a lack of time (drop out). The value was 
assigned during data processing only. 

• Logically not applicable (6): The respondent answered a preceding filter question in a 
way that made the following dependent questions not applicable to him or her. This 
value was assigned during data processing only. 

9.4 Interim Data Products 

Building the SITES 2006 international database was an iterative process in which the lEA 
DEC provided the consortium and the NRCs with a new version of data files whenever a 
major step in data processing was completed. This process guaranteed that the NRCs had 
a chance to review their data and to run their own plausibility and statistical checks to 
validate the data. The data products that were sent out by the lEA DEC to the consortium 
and to each NRC included both data files as well as data summaries. All interim data 
were made available to the consortium whereas each education system received its own 
data only. 

The first version was sent to the consortium and the NRCs as soon as the data 
could be regarded as “clean” relative to identification codes and linkage issues. Once 
weights and information-facilitating variance estimation became available (see Chapter 
10), these were added to the principal and teacher data files. The lEA DEC sent out a 
third version once the majority ofbackground cleaning issues had been resolved and final 
updates to the data files implemented. These files, which enabled the NRCs to replicate 
the results presented in the first draft chapters of the international report, were also used 
in an initial international database (IDB) training held by lEA DEC staff during the third 
NRC meeting in Frascati, Italy, in April 2007. 

Summary tables containing unweighted univariate statistics for all questionnaire 
variables were provided for each participating education system. For categorical 
variables, which represent the majority of variables in SITES 2006, the percentages of 
respondents choosing each of the response options were displayed. For numeric or scale 
variables, various descriptive measures were reported. These included the minimum, 
the maximum, the mean, the standard deviation, the median, the mode, percentiles, 
and quartiles. For both types of variables, the percentages of missing information due 
to respondents omitting or not reaching a particular question were reported. These 
summaries were used for an in-depth review of the data at the international as well as the 
national level in terms of plausibility, unexpected response patterns, conspicuous profiles 
of an education system, and so on. 

9.5 Building the International Database (IDB) 

All interim data products were placed in the structure used during data entry: thus, 
one file for the principal data, one file for the ICT coordinator data, and one file for 
teacher data, and with each presented separately for each participating education system. 
However, both the principal and the ICT coordinator responses related to the school 
level and matched each other 1:1 (see Chapter 3). Because weights were calculated 
only once for each school (see Chapter 10), these two files were consequently merged 
to form a combined school-level data file comprising the responses from the principal 
as well as from the ICT coordinator. The teacher data-entry file contained responses 
from two independently sampled populations of mathematics and science teachers to 
simplify data entry and logistics. Because these teachers belonged to two independent 
populations and because the weighting for these two groups was done separately and 
independently (see Chapter 10), analysis of combined mathematics and science teacher 
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data was neither meaningful nor statistically possible. To reflect this situation, the interim 
data entry file for teachers was split into two files for the final international database, one 
for mathematics teacher data and one for science teacher data, with each including its 
own sampling weights. 

A second key difference between the interim data products and the draft and final 
(public use) databases is the fact that the former included one record for each sampled 
unit (school or teacher) even if the questionnaire was not returned or returned empty. 
The draft and final IDE included only those records that satisfied the SITES 2006 
sampling standards. Data from respondents who either did not participate or did not 
pass adjudication (e.g., because the within-school participation was not sufiicient) were 
removed at the final stage. 

For the draft and final IDE, the data cleaning at the lEA DEC ensured that 
information coded in each variable was, in fact, internationally comparable, that national 
adaptations were reflected appropriately in all concerned variables, that questions 
not internationally comparable were removed from the database (see Appendix D), 
and (eventually) that all entries could be successfully linked across levels. The IDE 
incorporated all national data files and was prepared in raw, SAS, and SPSS format. 

Following the data release policy agreed between each NRC and the lEA, a draft 
IDE that included data from all education systems was made available to each NRC in 
early January 2008 prior to the publication of the international report in spring 2008. 
The data release policy granted freedom to publish any national results at any time (as 
long as appropriate reference to the lEA was made), but obliged countries to delay the 
publishing of any analysis involving more than their own data (e.g., comparisons of 
education systems) until after publication of the international report. The public-use 
IDE was supplemented by full documentation, national context data (see Chapter 2), 
and a detailed user guide. 

The SITES 2006 IDE is a unique resource for policymakers and analysts. It 
contains data from representative samples of schools and Grade 8 mathematics and 
Grade 8 science teachers from 22 education systems around the world. The database is 
fully documented in the SITES 2006 User Guide (Erese & Garstens, 2009). 
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10.1 Overview 

This chapter provides detailed information on why and how sampling weights were 
calculated. It also gives an overview of the (un-weighted and weighted) participation 
rates of schools, teachers, principals, and ICT coordinators. The method of variance 
estimation used in SITES 2006 also is explained. Special characteristics of the samples 
for the participating education systems and details about sample implementation can be 
found in Appendix B. 

10.2 Within-school Participation Requirements 

When teacher participation rates within a school are small, the risk of having responding 
teacher(s) who are not typical of the entire body of teachers in the school strengthens. 
Within-school participation requirements were therefore set up to prevent possible non- 
response bias. 

A school was regarded as participating if: 

• It returned the principal questionnaire or the technical questionnaire and at least two 
teacher questionnaires (regardless of the subject domain); or 

• It returned at least 50% of the teacher questionnaires (regardless of the subject 
domain). 

A teacher from a participating school was considered as participating if he or she 
returned a questionnaire and provided a valid response to at least one of the questions. 
Similar requirements were set for principals and ICT coordinators. Schools where these 
participation requirements were not met were regarded as non-participating, and none 
of the questionnaires submitted from teachers, principals, or ICT coordinators of these 
schools was used for data analysis. 

10.3 Calculating Sampling and Estimation Weights 

The SITES 2006 sampling design required the use of sampling weights for data analysis. 
As described in Chapter 6, schools in different explicit strata had different selection 
probabilities, which made it more likely for large schools than small schools to be 
selected for SITES 2006. Teachers from different schools also had different selection 
probabilities. As a result, the participating schools and teachers did not always represent 
the same numbers of schools and teachers in the population. This fact had to be 
accounted for by the use of sampling weights, which correctly reflected the different 
selection probabilities. 
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In addition to this, the willingness of schools and teachers to participate in SITES 
2006 differed from one school to another. To reduce the potential bias introduced hy 
school and teacher non-response, sampling weights had to he adjusted for both school 
and teacher non-response. 

This situation led to two distinct sets of sampling weights for SITES 2006: (i) 
a final school weight consisting of a basic school weight and a school non-response 
adjustment factor; and (ii) a final teacher weight consisting of the final school weight just 
mentioned, a basic teacher weight, and a teacher non-response adjustment factor. 

All analyses in the SITES 2006 international report were conducted using these 
weights. (All secondary analyses of the SITES 2006 database should be performed with 
weighted data only.) These weights and their components are described in the following 
sections. 

10.3.1 Basic School Weight (wgtfaci) 

The basic school weight, defined as the inverse of the school selection probability, is 
labeled wgifacl. For each school k in explicit stratum i, the basic weight is given by: 

wgtfaclik^^ , 

where Ni is the number of schools in the explicit stratum i, and n, is the number of 
sampled schools from that stratum. 

10.3.2 School Non-response Adjustment (wgtadji) 

Given that some schools refused to participate in SITES 2006, it was necessary to adjust 
the above basic weights to account for the loss of sample size due to unit non-response. 
In an attempt to minimize potential bias due to non-response, adjustment factors were 
calculated within distinct groups of schools, called non-response adjustment cells. These 
cells were formed at the implicit stratum level (see Section 6.4.3). When there was no 
implicit stratification, these cells were built at the explicit stratum level (see Section 
6.4.1). When there were fewer than five participating schools within a cell, or when the 
adjustment factor was larger than 2, cells were collapsed with other cells that had similar 
characteristics until the number of participating schools within the resulting cell was five 
or higher and the adjustment factor less than 2. 

The school non-response adjustment factor wgtadji is defined as follows for each 
responding school k within non-response adjustment cell j in stratum i: 

s rl r2 nr 
nij + riij +riij +tiij 

s rl r2 •' 

with n^ij being the number of originally sampled schools that participated, n^^ij and n^^ij 
the number of first and second replacement schools (see Section 6.6), respectively, that 
participated, and n'*''y the number of schools that did not participate in response group; 
of explicit stratum i. 

Ineligible schools (no Grade 8 students), closed schools, and schools belonging 
to the exclusion categories were ignored in the non-response adjustment cells. 

10.3.3 Final School Weight (schwgt) 

The final school weight (schwgt) is the product of wgtfaci and wgtadji as indicated 
here: 

schwgt-, - wgtfaci-, X wgtadji -, 

where each participating school k is in response group; of explicit stratum i. This weight 
was used for all data analyses involving variables relating to the school principal and the 
IGT coordinator and presented in the international report. 
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w.3.4 Second Stage Teacher Weight (wgtfaci) 



Teacher weight factors were calculated separately and independently for mathematics 
teachers and for science teachers. The following account explains the process for 
mathematics teachers only given that the calculation of science teacher weights is 
performed in the same way. 

The basic teacher weight (second stage teacher weight) is defined by the inverse 
of the selection probability of the teacher within the sampled school. For each sampled 
teacher I of school k, the basic teacher weight is given by: 



wgtfacliki - 






where M,fc is the number of mathematics teachers in the school k (that were not excluded 
before sampling) and mik is the number of sampled mathematics teachers in that 
school. 

10.3.5 Teacher Non-response Adjustment (wgtadji) 

Due to small within-school sample sizes^ the research consortium deemed it preferable 
not to implement the teacher non-response adjustment at the school level, but instead 
to implement it at the school non-response adjustment cell level. In other words, school 
and teacher non-response adjustments were performed at the same level (see Section 
10.3.2 above). 

Within each adjustment cell j of explicit stratum i, the teacher non-response 
adjustment factor wgtadji is given by: 

'Ewgtfaclijki 

wgtadjlijkl^ ' 

'Zwgtfaclijkl 

participants 

where wgtfadijkl is the basic teacher weight for teacher I in school k of explicit stratum 
i; and the numerator is summed over all sampled teachers whereas the denominator is 
summed over participating teachers only. 

Teachers who had left school permanently after sampling, or teachers who were 
found to be part of the group of excluded teachers, were omitted from this equation. 
Some systems reported teachers being temporarily absent, for example on maternity 
leave or sick leave. To obtain correct estimations of the teacher population size, these 
teachers were included in the teacher non-response adjustment calculations. 

10.3.6 Final Teacher Weight (totwgt) 

The final teacher weight, denoted as totwgt (total weight), is the product of four weight 
factors, which are also included in the international database: wgtfaci, wgtadji, wgtfaci, 
and wgtadji. The weight totwgt was used for all data analyses involving teacher-related 
variables in the SITES 2006 international report. 

Note that it is neither meaningful nor recommended to group together data from 
mathematics teachers and science teachers for analytical purposes. The combined data 
set cannot represent a population of teachers who teach “mathematics, science, or both.” 
In many SITES 2006 systems, some teachers taught both mathematics and science. 
If both data sets had been (or are) combined, these teachers would have been over- 
represented. Some teachers filled in SITES 2006 questionnaires for mathematics and for 
science, so their input to the combined data was doubled. Some other teachers taught 
mathematics and science, but were only sampled for one of the two subjects. Being part 
of two teacher sampling frames, they had a higher probability of being selected than did 
teachers who taught only one subject. Consequently, analysis for the mathematics and 
the science teachers had to (and must always) be done separately. 
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70.5.7 System-specific Issues 

• Italy: In Italy, mathematics and science classes are taught simultaneously hy the 
same teacher. Therefore, the teacher sample selection and the weight calculation 
for SITES 2006 were not done separately hy subject. This meant that the totwgt 
variable added up to the estimated number of “mathematics and science” teachers 
for this system. However, half of the teachers were asked to fill in the questionnaire 
with regards to teaching mathematics, while the other half were asked to do it with 
regards to teaching science. The data for both groups were analyzed separately in the 
international report. 

• Russian Federation: In Russia, regions were selected as a first sampling stage. 
Therefore, each school weight was multiplied by a region weight factor that reflected 
the probability of selecting any one region. 

10.4 Calculating School and Teacher Participation Rates 

Weighted and un-weighted participation rates were calculated at school and teacher 
levels to facilitate the evaluation of data quality and the risk of potential biases due to 
higher levels of non-response. 

10.4.1 Un-weighted Participation Rates 

The un-weighted school participation rate is given by dividing the total number of 
participating schools (as defined by the requirements described in Section 10.2) by the 
number of sampled and eligible schools. This rate was calculated twice: once based on the 
originally sampled schools only, and once in order to include the replacement schools. 

The un-weighted teacher participation rate is given by the number of teachers 
in the participating schools divided by the number of teachers who were sampled and 
eligible in the participating schools. This calculation was performed separately for the 
mathematics and the science teachers. 

The un-weighted participation rate for principals is given by the number of 
principals in the participating schools divided by the number of participating schools. The 
un-weighted participation rate for ICT coordinators was calculated in a similar manner. 
Table 10. 1 shows the un-weighted participation rates for all participating systems. 

10.4.2 Weighted Participation Rates 

The participation rates were also calculated using the school and teacher weights 
described above. In general, the difference between the un-weighted and the weighted 
participation rates was small. Table 10.2 shows the weighted participation rates for all 
participating systems. 
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Table 10.1: Un-weighted Participation Rates in SITES 2006 



Education System 


School 
Participation 
Rate before 
Replacement 
(%) 


School 
Participation 
Rate after 
Replacement 
(%) 


Math 

Teacher 

Participation 

Rate 

(%) 


Science 

Teacher 

Participation 

Rate 

(%) 


Principals 

(%) 


ICT Coordinators 
(%) 


Alberta, Canada 


73 


75 


83 


80 


67 


57 


Catalonia, Spain 


85 


89 


94 


91 


96 


92 


Chile 


77 


88 


93 


93 


94 


95 


ChineseTaipei 


97 


100 


97 


97 


99 


100 


Denmark 


55 


66 


80 


80 


80 


74 


Estonia 


53 


53 


78 


76 


85 


73 


Finland 


67 


74 


86 


83 


86 


90 


France 


52 


63 


85 


82 


89 


82 


Hong Kong SAR 


67 


71 


76 


79 


76 


90 


Israel 


92 


94 


85 


86 


85 


86 


Italy 


76 


92 


89 


89 


90 


94 


Japan 


79 


99 


97 


96 


100 


100 


Lithuania 


72 


73 


90 


87 


87 


90 


Moscow, Russian Federation 


100 


100 


99 


99 


100 


100 


Norway 


53 


61 


80 


81 


75 


75 


Ontario, Canada 


82 


82 


93 


94 


97 


86 


Russian Federation 


99 


99 


99 


99 


99 


100 


Singapore 


100 


100 


98 


98 


92 


99 


Slovak Republic 


90 


98 


97 


96 


97 


96 


Slovenia 


91 


91 


93 


91 


90 


94 


South Africa 


90 


91 


90 


90 


99 


91 


Thailand 


80 


93 


97 


96 


96 


96 



10.5 Meeting the SITES 2006 Sampling Standards 

Calculation of the participation rates was followed by an assessment of the quality of 
the sampling implementation. Based on the school and teacher participation rates, each 
system was assigned to a category indicating the quality of the sample implementation. In 
some systems, issues other than response rate requirements made it necessary to advise 
the readers of the international report or the users of the international database (IDB) 
that they should interpret the study results with caution. 
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Table 10.2: Weighted Participation Rates in SITES 2006 
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Table 10.2: Weighted Participation Rates in SITES 2006 



Education System 


School 
Participation 
Rate before 
Repiacement 
(%) 


Schooi 
Participation 
Rate after 
Replacement 
(%) 


Math 

Teacher 

Participation 

Rate 

(%) 


Science 

Teacher 

Participation 

Rate 

(%) 


Principals 

(%) 


ICT Coordinators 
(%) 


Alberta, Canada 


72 


75 


84 


81 


65 


53 


Catalonia, Spain 


84 


89 


94 


91 


96 


92 


Chile 


76 


87 


94 


93 


94 


94 


ChineseTaipei 


96 


100 


97 


97 


99 


100 


Denmark 


55 


66 


79 


78 


79 


73 


Estonia 


53 


53 


78 


76 


85 


74 


Finland 


68 


76 


86 


84 


86 


90 


France 


50 


61 


86 


83 


88 


81 


Hong Kong SAR 


67 


71 


76 


80 


76 


90 


Israel 


91 


93 


85 


88 


85 


87 


Italy 


76 


92 


90 


90 


90 


94 


Japan 


79 


99 


97 


96 


100 


100 


Lithuania 


70 


72 


91 


88 


88 


89 


Moscow, Russian Federation 


100 


100 


99 


99 


99 


100 


Norway 


50 


60 


80 


81 


75 


73 


Ontario, Canada 


82 


82 


94 


94 


96 


86 


Russian Federation 


99 


99 


99 


99 


99 


100 


Singapore 


100 


100 


97 


96 


92 


99 


Slovak Republic 


89 


98 


97 


96 


97 


96 


Slovenia 


91 


91 


93 


92 


90 


94 


South Africa 


90 


91 


90 


90 


99 


90 


Thailand 


79 


92 


98 


97 


97 


96 



10.5.1 Participation Categories 

At the school level, four categories were defined.^ Meeting the criteria based on either 

the un-weighted or the weighted participation rates was deemed sufficient to reach a 

specific participation category. 

• Category 1 : The school data were fully adjudicated if the school participation rate 
before replacement was at least 85%. 

• Category 2: The school data were classified in Category 2 if the school response rate 
before replacement was at least 50% and if the response rate after replacement was at 
least 85%. 

• Category 3: The school data were classified in Category 3 if the school response rate 
before replacement was at least 50% and if the response rate after replacement was at 
least 70%. 

• Category 4: The school data were classified in Category 4 if the system failed to reach 
either a response rate of 50% before replacement or a response rate of 70% after 
replacement (or both). 



2 Note that Categories 1 and 2 are similar to the TIMSS and PIRLS sampling standards. 
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Unlike school-level data^ teacher-level data are affected hy two factors: school 
participation and teacher participation. In SITES 2006, the teacher-level participation 
categories were assigned independently for mathematics teachers and for science 
teachers. The categories were defined as follows: 

• Category 1 : The teacher data were fully adjudicated if the school data were classified 
in Category 1 and the mathematics (science) teacher response rate was at least 85%. 

• Category 2.- The teacher data were classified in Category 2 if the school data were 
classified in Category 2 and the mathematics (science) teacher response rate was at 
least 85%. 

• Category 3: The teacher data were classified in Category 2 if the school data were 
classified in Category 1, 1, or 3 and the mathematics (science) teacher response rate 
was at least 70%. 

• Category 4: The teacher data were classified in Category 4 if the school data were 
classified in Category 4 or if the participation rate of the mathematics (science) 
teachers was under 70%. 

Relative to this definition, the teacher-level participation categories could only he 
equal to or worse than the school-level categories in each system. However, the teacher- 
level participation rates never led to downgrading a system from a school-participation 
category to a lower teacher-level category because, in general, the within-school 
participation was higher than the school participation. Table 10.3 shows the participation 
categories for all SITES 2006 systems. 

The categorization affected the presentation of the data in the SITES 2006 
international report. Systems that fell into Categories 2 and 3 were assigned footnotes. 
Systems in Category 4 were reported in separate tables or graphs because of doubts over 
whether their data were comparable to the data of the other systems. 

10.5.2 System-specific Issues 

In some systems, parts of the survey administration differed slightly from the SITES 
2006 standard procedures: 

• Israel: In Israel, independent orthodox schools did not participate in SITES 2006. As 
a result, a substantial part of the school population was not represented. This situation 
was explained in a footnote in the teacher tables and graphs of the international 
report. 

• Japan: In Japan, teacher participation data were collected after survey administration. 
Because of the likelihood of the participation rates and the non-response adjustments 
being a little inaccurate as a result of this procedure, this situation was footnoted in 
the teacher tables and graphs. 

• Alberta, Canada: In the province of Alberta, Canada, less than 70% of the school-level 
questionnaires in the participating schools were returned. Because this return rate 
had the potential to slightly bias the school-level results, this situation was explained 
in a footnote in the tables and graphs containing school-level data. 

• Alberta (Canada), Lithuania, Russian Federation (including Moscow), South Africa, 
and Thailand: In these systems, some or all schools did not correctly select the target 
class according to the definition of target class (see Section 6.7). The lEA Technical 
Executive Group (TEG) therefore decided that the teacher data from these systems 
should be reported in separate tables or graphs. 
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Table 10.3: Participation Categories in SITES 2006 



Education System 


Participation Category 


Alberta, Canada 


3 


Catalonia, Spain 


1 


Chile 


2 


Chinese Taipei 


1 


Denmark 


4 


Estonia 


4 


Finland 


3 


France 


4 


Hong Kong SAR 


3 


Israel 


1 


Italy 


2 


Japan 


2 


Lithuania 


3 


Moscow, Russian Federation 


1 


Norway 


4 


Ontario, Canada 


3 


Russian Federation 


1 


Singapore 


1 


Slovak Republic 


1 


Slovenia 


1 


South Africa 


1 


Thailand 


2 



10.6 Estimating Sampling Variance 

In SITES 2006, the sampling design involved stratified multi-stage cluster sampling. In 
this kind of design, the standard errors of the population estimates cannot he estimated hy 
using simple textbook formulae. In SITES 2006, the standard errors were estimated using 
the jackknife repeated replication technique (JRR) based on established procedures in 
TIMSS 2003 (Gonzalez, Galia, Arora, Erberber, & Diaconu, 2004). 

The general use of JRR entails systematically assigning pairs of schools to 
sampling zones, and randomly selecting one of these schools to have its contribution 
doubled and the other to have its contribution zeroed, so as to construct a number of 
“pseudo-replicates” of the original sample. The statistic of interest is computed once 
for the original sample and once again for each pseudo-replicate sample. The variation 
between the estimates for each of the replicate samples and the original sample estimate 
is the jackknife estimate of the sampling error of the statistic. 

10.6.1 Constructing Sampling Zones for Variance Estimation 

Application of the JRR technique used in SITES 2006 necessitated pairing the sampled 
schools and assigning them to a series of groups known as sampling zones. This was done 
by working through the list of sampled schools in the order in which they were selected 
and assigning the first and second schools to the first sampling zone, the third and fourth 
schools to the second zone, and so on. 
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In total, 100 zones were used. When 100 zones had been completed, the process 
was continued hy assigning the next pair of schools to the first sampling zone, the next 
pair to the second sampling zone, and so on. In an ideal case of 400 participating schools, 
each zone would have consisted of four schools. Usually, some zones consisted of fewer 
than four schools. In systems with large school samples, some zones consisted of more 
than four schools. 

Paired schools had to belong to the same explicit stratum. Situations involving an 
odd number of schools in an explicit stratum translated into having one zone with only 
one school assigned. In practice, this meant that the variance contribution of that zone 
came from the difference between that school mean and the system mean by opposition 
to the difference between the schools within a zone. However, because of the repeated 
assignment of schools to the pairs described above, zones with only one school were 
very rare. 

10.6.2 Computing the Sampling Variance Using the JRR Method 

The JRR algorithm used in SITES 2006 assumes that there are 100 sampling zones within 
each system. To compute a statistic t from the sample for a system, the formula for the 
JRR variance estimate of the statistic t is given by the following equation: 

100 

VarjRR (t) - 2[t(/f,) - t(S)]^ 

h=l 

The term t(S ) corresponds to any weighted or un-weighted statistic for the whole 
sample; the element t(jJ denotes the same statistic using the jackknife replicate. This 
is computed using all cases except those in the zone of the sample. If there are two 
units in the zone, all cases associated with one randomly selected school in the zone 
are removed, and the elements associated with the other unit in the zone are included 
twice. If there are four units in the zone, all cases associated with the two randomly 
selected schools in the zone are removed, and the elements associated with the other two 
units in the zone are included twice. 

The computation of the JRR variance estimate for any statistic in SITES 2006 
required the computation of the statistic 10 1 times for any given system: once to obtain 
the statistic for the full sample, and 100 times to obtain the statistics for each of the 
jackknife replicates (J^). 

10.6.3 System-specific Issues 

Some systems had a somewhat different design. 

• Russian Federation: In the Russian Federation, a design with one additional sampling 
stage was implemented. First, a sample of regions was selected with selection 
probabilities proportional to size of the region. Second, schools were selected within 
these regions (see Appendix B for details). For variance estimation, sampled regions 
were paired. In regions that were large enough to be selected with certainty, the 
sampled schools were paired. 

• Singapore: A census of schools was conducted in Singapore. 

• Alberta (Canada) and Finland: In the Province of Alberta, Canada, and in Finland, 
some of the schools were selected for SITES 2006 with certainty. The assignment 
of teachers to variance strata was adapted to the fact that these schools were part of 
every possible school sample. This process involved assigning paired teachers instead 
of paired schools to the variance strata. 
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10.7 Quality Control 

Various measures were taken to ensure the correctness of the sampling weights, 
participation rates, and variance estimates. Among other checks, the following were 
performed for all sets of weights in the participating systems. 

• At the school level, the sum of the final school weights was checked to ensure it added 
up to the estimated number of eligible schools in the participating system, that is, the 
number of schools in the sampling frame minus the sum of wglfacl of any ineligible 
schools. 

• For the teacher weights, the following checks were performed: 

- The sum of wgtfac2 of the sampled teachers represented the number of teachers 
(that were not excluded before sampling) in the schools. 

- The sum of the product {wgtfac2 x wgtadj2) for the participating teachers added 
up to the sum of wgt^acl of the sampled eligible teachers. 

• For the mathematics teachers, the variable totwgt had to add up to the estimated 
number of SITES 2006-eligible mathematics teachers in the participating system. 
For the science teachers, totwgt had to add up to the estimated number of SITES 
2006-eligible science teachers in the participating system. A check was conducted to 
ensure these estimates were plausible according to the available information. 

These quality criteria were always met. As a final quality check, the weights were 
calculated independently by both authors of this chapter. No deviations were found. 

Reference 
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achievement in mathematics and science. In M. O. Martin, I. V. S. Mullis, & S. J. Ghrostowski 
(Eds.). (2004). TIMSS 2003 technical report (pp. 275-307) Boston, MA: TIMSS & PIRLS 
International Study Genter. 
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11. Scale and Indicator Construction 
for the School and Teacher Levels 

Nancy Law 
Angela Chow 
Willem Johan Pelgrum 



11.1 Overview 

In quantitative studies, scale and composite indicators are often preferred to indicators 
derived from single-item responses because they provide improved stability and 
reliability for important constructs. This chapter reports on the design and development 
of scale indicators used in the core component of the teacher questionnaire and the two 
school-level questionnaires. 

Confirmatory factor analysis (CFA) is widely recognized as a rigorous statistical 
technique to use to construct measurement models for confirming or disproving 
hypothesized underlying latent variable structures (Byrne, 1998). CFA has been used 
extensively in studies across different fields, such as psychology, marketing, and career 
counseling (e.g., Byrne, 1989; Harvey, Billings, & Nilan, 1985; Kumar & Sashi, 1989; 
Marsh, 1985; Thacker, Fields, & Tetrick, 1989). This methodology was adopted in the 
pre-pilot and field trial phases involving the development of the key indicators in the 
teacher questionnaire because it provides a better estimate of scale quality, which is 
important at the instrument design stage. A brief description of the teacher questionnaire 
data-scaling methodology is reported in Section 11.2. The quality indicators from the 
CFA for the field trial teacher questionnaire data are reported in the section following. 

For the main study data, Cronbach’s alpha reliability scores were reported for scale 
indicators in both the teacher and school questionnaires in order to reflect the quality of 
the indicators reported in the international report. These are reported in Section 11.4 
onwards. 

11.2 Scaling Methodology 

The first step in constructing a CFA model is to determine the specific link between the 
construct and its latent factors. Normally, exploratory factor analysis is conducted on 
data collected from the items pertaining to the construct to identify the likely number of 
latent factors and their constituent items (joreskog & Sorbom, 1993). Based on findings 
from the exploratory factor analysis, CFA can be conducted to check on the robustness 
of the factor model. Goodness of fit statistics to evaluate the robustness of a CFA model 
commonly include the following: root mean square error of approximation (RMSEA), 
developedby Steiger (1990); comparative fix index (CFI), developed by Bender ( 1990); 
and non-normed fit index (NNFI), developed by Bender and Bonett ( 1980). A model 
with a RMSEA lower than 0.1 and both a NNFI and a CFI larger than 0.9 is generally 
taken as statistically validated (Diamantopoulos & Siguaw, 2000; Kelloway, 1998). 

For each latent factor, the factor loading of its component items should be 
checked to see if there is (are) any dominating item(s). If there is no dominating item, 
the indicator score of a latent factor can be taken as the mean of its respective component 
items; if there is (are) dominating item(s), the indicator score is the factor score. 
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CFA is a more demanding method and requires more time and resources to 
implement. Scale indicators are often constructed based on prior theoretical or empirical 
studies without being validated through CFA. Under such circumstances, reliability is 
taken as a measure of scale quality. lEA’s Technical Standards f or lEA Studies (Martin, Rust, 
& Adams, 1999) recommends annotating and interpreting with caution any reliability 
below 0.7. In order not to make the questionnaire overly long, some of the scales ended 
up with only three items, which makes it difficult to achieve such a high reliability. Some 
commentators argue that a Cronbach’s alpha value above 0.5 is satisfactory (see, for 
example, Nunnally, 1978). In SITES 2006, a reliability of 0.5 or above was adopted as a 
“marginally” acceptable quality measure for a scale indicator. 

The CFA method was adopted in the analysis of the pilot test and field trial data for 
the construction of a number of scale indicators in the teacher questionnaire. The details 
of this process are reported in Section 1 1.3. It is important to point out that the validity 
of scale indicators as psychometric constructs can differ across countries/ systems due to 
socio-cultural, linguistic, and other contextual differences. Hence, it is desirable to check 
the quality of the scale indicators for each system to ensure that these indicators can be 
used for international comparison. Because of the small sample size of the field trial data 
for each participating system, the CFA was conducted on the entire international data 
set. In finalizing the indicators for use in the international report of the SITES 2006 main 
study, Cronbach’s alpha reliability scores for each of the indicators were computed for 
each participating system to ensure that the scale indicators were valid at the system level 
and could thus be used for comparative purposes. 

11.3 CFA Results for Potential Scale Indicators from Teacher Questionnaire 
Field Trial Data 

Only 17 of the 22 systems participating in the main study took part in the field trial. 
About 1,850 completed teacher questionnaires were collected. Exploratory factor 
analyses were conducted on data collected from the field trial on questions pertaining to 
the following seven sets of scale indicators: curriculum goal orientation, teacher practice 
orientation, student practice orientation, impact of ICT use on teachers themselves 
and on students, vision for ICT use in the future, and the presence of a community of 
practice in the school. Items for these indicators were developed based on the theoretical 
and empirical considerations described in Chapter 4. The CFA was conducted to ensure 
that the designed questions yielded scales with conceptual and statistical validity. We 
note here that the CFA was conducted on the pooled set of data collected from both the 
mathematics and the science teachers because the research consortium considered that 
these constructs were unlikely to have different factor structures for the two populations 
of teachers. Table 11.1 provides a summary of the CFA results. 

Based on the RMSEA, CFI, and NNFI indices for each set of scale indicators 
listed in Table 1 1.1, it is clear that all the scales met the required statistical requirements 
for validation. In order to reduce the length of the questionnaire, we decided to reduce 
the number of items in each of the pedagogical orientation indicators to three. Hence, 
for those indicators with more than three items, we performed judgmental and statistical 
examinations to eliminate the items with, for example, lower correlation with the other 
items in the same scale. 

It is important to note here that all three of the pedagogical orientation scales 
(curriculum goals, teacher practice, and student practice) yielded four factors that had 
good conceptual alignment with one another. However, the analysis of the final main 
study data found substantial differences across systems in terms of the factor structure 
for these three scales. A three-factor structure was therefore found to be more satisfactory 
for the purpose of international cross-system comparisons. This structure is reported in 
detail in a later section. 
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Table 11.1: Summary of the Key CFA Results for Seven of the Scales in the Field Trial Teacher 
Questionnaire 



Scale Label 
(Field Trial 
Question 
Number) 


N 


RMSEA 


CFI 


NNFI 


Factors 


Field Trial Item 

Number 

Reliability 


Curriculum goal 


1,434 


0.083 


0.96 


0.95 


1. Traditionally important goals 


A, B, C, D (0.61) 


orientation (T8) 










(academic achievement) 














2. Student-centered goals 


E,F,G(o.59) 












3. Goals about inquiry 


H,I,J (0.75) 












4. Goals about connectedness 


K, L, M (0.78) 


Teacher practice 


1,418 


0.079 


0.96 


0.94 


1. Traditionally important instruction 


A, B, C, D (0.61) 


orientation (T15) 










2. Student-centered pedagogy 


E, E, G (0.71) 












3. Support collaborative inquiry 


H,I,J (0.79) 












4. Support liaison for connectedness 


K, L, M (0.73) 












in learning 




Student practice 


1,339 


0.062 


o.g6 


0.95 


1. Traditionally important learning 


A, B, C, D (0.60) 


orientation (T17) 




(S8 

deleted) 






2. Student-centered learning 


E, E, G (0.64) 










3. Self-directedness/inquiry 


l,J, K,L(o.74) 












4. Collaboration and learningfrom others 


M,N, 0(0.71) 


ICT impacts on 


1,392 


0.049 


0.99 


0.99 


1. Empowerteaching 


A, B (0.85) 


teacher (Tig) 










2. Monitor student 


C, D (0.86) 












3. Collaboration 


E, F (0.75) 












4. Work easier 


G, H (0.79) 












5. Negative 


l,J (0.76) 


ICT impacts on 


1,346 


0.11 


0.97 


0.96 


1. Traditionally important 


A, B (0.86) 


students (T20) 










2. Collaborative inquiry 


E, F,G (0.90) 












3. ICT 


H, 1 (0.70) 












4. Negative 


K, L (0.89) 


Vision for ICT use 


1,402 


0.078 


0.97 


0.96 


1. Traditionally important instruction 


A, B (0.43) 


in nearfuture (T35) 
























2. Student-centered 


C, J (0.46) 












3. Inquiry 


D, E, E (0.78) 












4. Collaboration 


H, 1 (0.76) 


Community of 


1.324 


0.069 


0.97 


0.96 


1. School vision (T39) 


A, B, C, D, E (0.88) 


practice (T39-42) 










2. Decisionmaking (T40) 


A, B, C, D, E (0.85) 












3. Professional collaboration (T41) 


A, B, C, D, E (0.73) 












4. Support to teacher (T42) 


A, B, C, D, E (0.80) 



In addition to the indicators listed in Table 11.1, two further sets of scale 
indicators on “teacher planning” and “teacher belief” were included in the field trial 
as explanatory indicators. However, these indicators showed low correlation with the 
pedagogical orientation indicators, and no significant statistical difference in their basic 
descriptive statistics emerged across systems. These two sets of scales thus had less value 
as explanatory indicators and were discarded from the teacher questionnaire for the main 
study in order to reduce the length of the instrument. 
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During their second meeting, the NRCs carefully reviewed the item statistics and 
analysis results for the exploration of scale indicators. In addition to deleting unsatisfactory 
or non-informative items, the NRCs revised the wording of some questions to improve 
clarity. 

11.4 Scale Indicators in the Core Component of the Main Study Teacher 
Questionnaire 

As mentioned earlier, we needed to ensure that the reliabilities of the scale indicators 
were acceptable when computing these separately for each of the participating systems 
in the final analysis and the reporting of the main study. The reliabilities for the various 
indicators are summarized in Tables 11.2 to 11.16. As is evident, most of the indicators 
showed good reliabilities across all of the participating systems. The main exception 
was with the three sets of pedagogical orientation indicators, which showed large 
differences in their reliability scores for different systems. In addition, a three-factor 
model showed higher consistency across systems than a four-factor model found relative 
to an analysis of the entire set of international field-trial data. The three-factor model still 
comprised the two factors with items for the traditionally important and connectedness 
orientations. However, the third factor comprised all of the items in the student-centered 
and collaborative-inquiry orientations. We therefore decided that a three-factor model 
would best suit the final analysis (see also Section 4.2.2). Based on the content of the 
items, those of us attending the third NRC meeting labeled this factor lifelong learning. 
We also established that the reliability of the indicator for lifelong learning curriculum goal 
orientation across systems would be more consistent if we included only four rather than 
six items in the computation. 

In addition to considering the reliability coefficients reported for each participating 
education system, we computed a cross-system reliability coefficient, called the 
“international alpha on adjudicated systems,” on equally weighted data from those 
systems that satisfied the sampling standards and adhered to the administration 
procedures. These Cronbach’s alpha values are included in this technical documentation 
to provide an additional aid to interested readers as well as to researchers wishing to 
undertake secondary analysis to assess the reliability and validity of scale indicators 
for cross-system comparisons. Because of potential item bias in those systems where 
non-response might not be random or completely random, that is, potentially related 
to the variable of interest, systems that did not satisfy the sampling standards or did 
not adhere to the final teacher sampling stage were not considered in the computation 
of international alpha. For the same reason, the international report does not include 
international scale score averages or means. 

11.4.1 Curriculum Coal Orientation 

Question 8 of the teacher questionnaire asked : “In your teaching of the target class in this 
school year, how important is it for you to achieve the following goals?” Respondents 
were asked to select a response from four choices — “not at all,” “a little,” “somewhat,” and 
“very much” — placed on a Likert-type scale. Responses to this question were used to 
provide indicators for the curriculum goal orientation. Three indicators were computed: 
traditionally important (8B, 8E, and 8K), lifelong learning (8D, 8G, 8H, and 81), and 
connectedness (8C, 8J, and 8M). 

The reliabilities of these indicators are shown in Table 11.2. We can see that for 
lifelong learning and connectedness, the reliabilities for all the systems were quite satisfactory 
(above 0.5) in both subjects, with lifelong learning being the highest, exceeding 0.6 in 
every case. For the traditionally important indicator, a number of the systems’ reliabilities 
fell below 0.5, but were still above 0.4 in every case. 



SCALE AND INDICATOR CONSTRUCTION FOR THE SCHOOL AND TEACHER LEVELS 



Table 11.2: Reliabilities for the Curriculum Goal Orientation Indicators for both Mathematics 
Teachers and Science Teachers 



Flags 


Education System 


Goal — Traditionally 
Important 


Goal — Lifelong 
Learning 


Goai- 


-Connectedness 


Cronbach’s Alpha 


No. of 


Cronbach’s Alpha 


No. of 


Cronbach’s Alpha 


No. of 






Math 


Science 


Items 


Math 


Science 


Items 


Math 


Science 


Items 




Catalonia, Spain 


0.607 


0.632 


3 


0.684 


0.699 


4 


0.672 


0.696 


3 


1 


Chile 


0435 


0.549 


3 


0.721 


0-755 


4 


0.648 


0.526 


3 




Chinese Taipei 


0.629 


0.671 


3 


0.745 


0.767 


4 


0.606 


0.697 


3 


2 


Finland 


0.491 


0.570 


3 


0.736 


0.677 


4 


0.598 


0.631 


3 


2 


Hong Kong SAR 


0.648 


0-577 


3 


0-759 


0.740 


4 


0.738 


0.645 


3 


4 


Israel 


0.439 


0.515 


3 


0.731 


0.710 


4 


0.725 


0.704 


3 


1 


Italy 


0.587 


0.609 


3 


0-745 


0-739 


4 


0.639 


0.663 


3 


1-3 


Japan 


0.584 


0.536 


3 


0.622 


0.636 


4 


0.657 


0.625 


3 


2 


Ontario, Canada 


0.479 


0-545 


3 


0.649 


0.710 


4 


0.588 


0.611 


3 




Singapore 


0.443 


0.542 


3 


0.798 


0.801 


4 


0.798 


0-733 


3 




Slovak Republic 


0.523 


0.518 


3 


0.672 


0.681 


4 


0.621 


0.626 


3 




Slovenia 


0.474 


0.517 


3 


0.687 


0.680 


4 


0.726 


0.684 


3 




International alpha on 


0.592 


o.6i8 


3 


0.772 


0.766 


4 


0.692 


0.672 


3 




adjudicated systems 





















Systems not satisfying participation standard or not adhering to survey administration procedure 



t, 2 


Alberta, Canada 


0.504 


0.459 


3 


0.711 


0.768 


4 


0.699 


0.682 


3 


# 


Denmark 


0.580 


0.604 


3 


0.647 


0.673 


4 


0.613 


0.564 


3 


# 


Estonia 


0.513 


0-539 


3 


0.665 


0.708 


4 


0.538 


0-559 


3 


# 


France 


0.481 


0.472 


3 


0-709 


0.649 


4 


0.607 


0.606 


3 


t, 2 


Lithuania 


0.586 


0.628 


3 


0-739 


0.760 


4 


0.692 


0.718 


3 


t 


Moscow, Russian 
Federation 


0.517 


0.485 


3 


0.696 


0-703 


4 


0.639 


0.636 


3 


# 


Norway 


0.532 


0-433 


3 


0.687 


0.679 


4 


0.651 


0-594 


3 


t 


Russian Federation 


0.496 


0.468 


3 


0.632 


0.687 


4 


0.658 


0.667 


3 


t 


South Africa 


0.614 


0.637 


3 


0.833 


0.796 


4 


0-730 


0.682 


3 


t, 1 


Thailand 


0.606 


0.635 


3 


0.785 


0.814 


4 


0.706 


0.723 


3 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 



n.4.2 Teacher Practice Orientation 

Question 14 of the teacher questionnaire asked: “In your teaching of the target class in 
this school year, how often do you conduct the following?” Respondents were asked to 
select a response from four choices — “never” “sometimes ” “often,” and “nearly always” — 
placed on a Likert-type scale. Responses to this question were used to provide indicators 
for the teacher practice orientation. Similar to the situation with the curriculum goal 
orientation, three sets of indicators were identified: traditionally important (14A, 14E, 
and 14G), lifelong learning (liB, 14C, 14D, 14F, 14H, and 14K), and connectedness (14I, 
14J, and 14L). 

Table 11.3 shows the reliahilities for these indicators. As is evident, the reliahilities 
for all the systems were satisfactory (above 0.5) for the lifelong learning and connectedness 
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indicators among the mathematics and the science teachers, with lifelong learning 
emerging as the best indicator (exceeding 0.6). Results varied across systems for the 
traditionally important indicator, with some systems falling helow 0.5 for both subjects. 
None of the reliabilities for systems meeting the standards for international comparison 
was lower than 0.4. 



Table 11.3: Reliabilities for the Teacher Practice Orientation Indicators for both Mathematics 
Teachers and Science Teachers 



Flags 


Education System 


TP 


— Traditionally 
Important 


TP — Lifelong 
Learning 


TP- 


-Connectedness 


Cronbach’s Alpha 


No. of 


Cronbach’s Alpha 


No. of 


Cronbach’s Alpha 


No. of 






Math 


Science 


Items 


Math 


Science 


Items 


Math 


Science 


Items 




Catalonia, Spain 


0.588 


0.633 


3 


0.752 


0.725 


6 


0-579 


0-579 


3 


1 


Chile 


0.724 


0.702 


3 


0.837 


0.838 


6 


0.698 


0.697 


3 




Chinese Taipei 


0.684 


0.669 


3 


0.800 


0.798 


6 


0.649 


0.680 


3 


2 


Finland 


0.463 


0.483 


3 


0.644 


0.673 


6 


0.570 


0.614 


3 


2 


Hong Kong SAR 


0.586 


0.541 


3 


0.729 


0.750 


6 


0.728 


0.687 


3 


4 


Israel 


0.611 


0.639 


3 


0.752 


0.811 


6 


0.679 


0.682 


3 


1 


Italy 


0-559 


0.582 


3 


0.804 


0.796 


6 


0.610 


0.571 


3 


1-3 


Japan 


0.431 


0-397 


3 


0.687 


0.692 


6 


0.657 


0.673 


3 


2 


Ontario, Canada 


0.517 


0-534 


3 


0.714 


0.738 


6 


0.659 


0.710 


3 




Singapore 


0.591 


0.596 


3 


0.760 


0.769 


6 


0.701 


0.660 


3 




Slovak Republic 


0.444 


0-459 


3 


0.754 


0.708 


6 


0.601 


0.605 


3 




Slovenia 


0.465 


0.516 


3 


0-735 


0.760 


6 


0.585 


0.648 


3 




International alpha on 
adjudicated systems 


0.586 


0.576 


3 


0.805 


0.815 


6 


0.712 


0.712 


3 



Systems not satisfying participation standard or not adhering to survey administration procedure 



t ,2 


Alberta, Canada 


0.604 


0.631 


3 


0.719 


0-735 


6 


0.638 


0.587 


3 


# 


Denmark 


0.401 


0.342 


3 


0.730 


0-777 


6 


0.681 


0.694 


3 


# 


Estonia 


0.401 


0.389 


3 


0.788 


0.811 


6 


0.563 


0-703 


3 


# 


France 


0.438 


0.397 


3 


0.638 


0.709 


6 


0.600 


0.650 


3 


t ,2 


Lithuania 


0.603 


0-535 


3 


0.786 


0.789 


6 


0.629 


0.712 


3 


t 


Moscow, Russian 
Federation 


0.383 


0.392 


3 


0.764 


0.796 


6 


0.659 


0.656 


3 


# 


Norway 


0.338 


0.408 


3 


0.650 


0.677 


6 


0.671 


0.617 


3 


t 


Russian Federation 


0.325 


0.272 


3 


0.709 


0.718 


6 


0.561 


0.632 


3 


t 


South Africa 


0.682 


0.669 


3 


0.827 


0.812 


6 


0.776 


0.756 


3 


t, 1 


Thailand 


0.566 


0.691 


3 


0.847 


0.850 


6 


0.657 


0.722 


3 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 
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77.4.5 Student Practice Orientation 

Question 16 of the teacher questionnaire asked: “In your teaching of the target class 
in this school yeaq how often do your students engage in the following activities?” 
Respondents were asked to select a response from four choices — “never” “sometimes” 
“often/’ and “nearly always” — placed on a Likert-type scale. Responses to this question 
were used to provide indicators for the student practice orientation. The initial design 
also had three sets of indicators: traditionally important (I6A, 16C, and 16H), lifelong 
learning {I 6 B, 16D^ 16E^ 16F^ 161, and 16J), and connectedness (I6G, 16K, and 16L). 

The reliabilities for the lifelong learning and connectedness indicators shown in 
Table 11.4 all had acceptable reliability values above 0.5. However, we could not find 
any combination of the three items “complete worksheets, exercises,” “work on the same 
learning materials at the same pace and/ or sequence,” and “answer tests or respond to 
evaluations” that would yield even a marginally acceptable scale for the traditionally 
important student practice orientation. This was because half of the systems had 
reliabilities lower than 0.4. On further inspecting the descriptive statistics for these three 
items, we found that the item “complete worksheets, exercises” had the highest overall 
mean and also the highest mean score across most of the participating systems. 

We also found that the other two items, “work on the same learning materials at 
the same pace and/ or sequence” (A) and “answer tests or respond to evaluations” (H), 
showed extremely low or even negative correlation with the item “complete worksheets, 
exercises” (C) among the countries that showed low reliabilities for this intended scale 
of three items. Based on these explorations, we selected the mean score for the item 
“complete worksheets, exercises” (C) for use as an indicator for traditionally important 
student practice in further analysis involving this concept in this study. 

77.4.4 Self-reported Impact of ICT Use on Teachers 

Question 19 of the teacher questionnaire asked: “To what extent do you agree that the 
use of ICT has had the following impacts on you?” Respondents were asked to select 
a response from four choices — “not at all,” “a little,” “somewhat,” and “a lot” — placed 
on a Likert-type scale. Responses to this question were used to provide indicators for 
the self-reported impact of ICT use on teachers. Table 11.5 and Table 11.6 contain the 
reliabilities for the scale indicators. All together, there were six indicators in this category, 
only four of which were scale indicators comprising two or more items: empower teaching 
(19B, 19D, and 19F), monitor students (19C and 19F), collaboration (19C and 19H), 
and negative impacts (19J, 19K, and 19L). The indicators for ICT skills (19A) and 
administrative efficiency ( 191) were single items only. 

As can be seen in Table 11.5 and Table 11.6, the reliabilities presented were all 
relatively high and greater than 0.5 in all cases for both the mathematics and the science 
teachers. 



99 



SITES 2006 TECHNICAL REPORT 



Table 11.4: Reliabilities for the Student Practice Orientation Scale Indicators for both 
Mathematics Teachers and Science Teachers 







SP- 


-Lifelong Learning 


SP- 


-Connectedness 


Flags 


Education System 


Cronbach’s Alpha 


No . of 


Cronbach’s Alpha 


Wo. 0/ 






Math 


Science 


Items 


Math 


Science 


Items 




Catalonia, Spain 


0.675 


0.656 


6 


0.639 


0.507 


3 


1 


Chile 


0.808 


0.802 


6 


0.812 


0.729 


3 




Chinese Taipei 


0-737 


0.746 


6 


0.714 


0.695 


3 


2 


Finland 


0.521 


0.634 


6 


0.479 


0.765 


3 


2 


Hong Kong SAR 


0.769 


0.791 


6 


0.843 


0.814 


3 


4 


Israel 


0-775 


0-793 


6 


0.703 


0-747 


3 


1 


Italy 


0-704 


0.727 


6 


0.669 


0.585 


3 


1-3 


Japan 


0.681 


0.672 


6 


0.814 


0.603 


3 


2 


Ontario, Canada 


0.741 


0-753 


6 


0.741 


0.638 


3 




Singapore 


0.787 


0.787 


6 


0-795 


0.833 


3 




Slovak Republic 


0.660 


0.631 


6 


0.562 


0.630 


3 




Slovenia 


0.614 


0.625 


6 


0.677 


0.651 


3 




International alpha on 
adjudicated systems 


0.748 


0.765 


6 


0.744 


0.717 


3 


Systems not satisfying participation standard or not adhering to su 


rvey administration procedure 


t,2 


Alberta, Canada 


0.679 


0.692 


6 


0.643 


0.709 


3 


# 


Denmark 


0.637 


0.726 


6 


0.506 


0.558 


3 


# 


Estonia 


0.648 


0.640 


6 


0.676 


0.696 


3 


# 


France 


0.643 


0.523 


6 


0-574 


0.556 


3 


t, 2 


Lithuania 


0.717 


0.757 


6 


0.685 


0.686 


3 


t 


Moscow, Russian 
Federation 


0.714 


0.701 


6 


0.677 


0.701 


3 


# 


Norway 


0.682 


0.632 


6 


0-595 


0.510 


3 


t 


Russian Federation 


0.682 


0.712 


6 


0.601 


0.671 


3 


t 


South Africa 


0.802 


0.810 


6 


0.833 


0.833 


3 


t,l 


Thailand 


0.816 


0.846 


6 


0.792 


0.796 


3 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 
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Table ll.S: Reliabilities for the Scale Indicators on Self-Reported Impact of ICT Use on 
Teachers (Mathematics) 



Flags 


Education System 


ICT Impact 
on Teacher- 
Empower Teaching 


ICT Impact 
on Teacher- 
Monitor Student 


ICT Impact 
on Teacher- 
Collaboration 


ICT Impact 
on Teacher- 
Negative Impact 






Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 




Catalonia, Spain 


o. 68 i 


3 


0.646 


2 


0.617 


2 


0.628 


3 


1 


Chile 


0.827 


3 


0.822 


2 


0.740 


2 


0.618 


3 




Chinese Taipei 


0.817 


3 


0.689 


2 


0.808 


2 


0.785 


3 


2 


Finland 


0-111 


3 


0.760 


2 


0.699 


2 


0.760 


3 


2 


Hong Kong SAR 


0.719 


3 


0.605 


2 


0.658 


2 


0.827 


3 


4 


Israel 


0.707 


3 


0.711 


2 


0.710 


2 


0.708 


3 


1 


Italy 


0.768 


3 


0.736 


2 


0.602 


2 


0.735 


3 


1-3 


Japan 


0.664 


3 


0.635 


2 


0.576 


2 


0.706 


3 


2 


Ontario, Canada 


0.783 


3 


0-793 


2 


0.694 


2 


0-733 


3 




Singapore 


0.806 


3 


0.748 


2 


0.787 


2 


0.788 


3 




Slovak Republic 


0.612 


3 


0.617 


2 


0.491 


2 


0.569 


3 




Slovenia 


0.752 


3 


0.710 


2 


0-747 


2 


0.722 


3 




International alpha on 


0.808 


3 


0.765 


2 


0.738 


2 


0.736 


3 



adjudicated systems 



Systems not satisfying participation standard or not adhering to survey administration procedure 



t,2 


Alberta, Canada 


0-775 


3 


0.704 


2 


0.702 


2 


0.806 


3 


# 


Denmark 


0.711 


3 


0.721 


2 


0.494 


2 


0-759 


3 


# 


Estonia 


0.729 


3 


0.700 


2 


0.654 


2 


0.679 


3 


# 


France 


0-733 


3 


0.789 


2 


0.591 


2 


0.636 


3 


t,2 


Lithuania 


0.811 


3 


0.676 


2 


0.710 


2 


0.729 


3 


t 


Moscow, Russian 
Federation 


0.693 


3 


0.536 


2 


0.764 


2 


0.693 


3 


# 


Norway 


0.672 


3 


0-793 


2 


0.624 


2 


0-747 


3 


t 


Russian Federation 


0.752 


3 


0.715 


2 


0-745 


2 


0.629 


3 


t 


South Africa 


0.931 


3 


0.854 


2 


0.910 


2 


0.853 


3 


t,l 


Thailand 


0.870 


3 


0.859 


2 


0.787 


2 


0.714 


3 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 
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Table 11.6: Reliabilities for the Scale Indicators on Self-reported Impact of ICT Use on 
Teachers (Science) 



Flags 


Education System 


ICT Impact 
on Teacher- 
Empower Teaching 


iCT Impact 
on Teacher- 
Monitor Student 


ICT Impact 
on Teacher- 
Collaboration 


ICT Impact 
on Teacher- 
Negative Impact 






Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 




Catalonia, Spain 


0.742 


3 


0.699 


2 


0.637 


2 


0.667 


3 


1 


Chile 


0793 


3 


0.780 


2 


0.749 


2 


0.572 


3 




Chinese Taipei 


0.791 


3 


0.781 


2 


0.717 


2 


0.772 


3 


2 


Finland 


0.796 


3 


0.783 


2 


0.721 


2 


0-753 


3 


2 


Hong Kong SAR 


0.716 


3 


0.638 


2 


0.717 


2 


0.827 


3 


4 


Israel 


0.744 


3 


0.784 


2 


0.730 


2 


0.718 


3 


1 


Italy 


0.783 


3 


0.667 


2 


0.640 


2 


0-704 


3 


1-3 


Japan 


0.642 


3 


0.558 


2 


0.696 


2 


0.749 


3 


2 


Ontario, Canada 


0.780 


3 


0.758 


2 


0.601 


2 


0.712 


3 




Singapore 


0.674 


3 


0.741 


2 


0.707 


2 


0.765 


3 




Slovak Republic 


0.603 


3 


0.604 


2 


0.629 


2 


0.500 


3 




Slovenia 


0.762 


3 


0.768 


2 


0-747 


2 


0.706 


3 




International alpha on 
adjudicated systems 


0.792 


3 


0.772 


2 


0.729 


2 


0.726 


3 



Systems not satisfying participation standard or not adhering to survey administration procedure 



t,2 


Alberta, Canada 


0.774 


3 


0.768 


2 


0.741 


2 


0.731 


3 


# 


Denmark 


0-739 


3 


0.814 


2 


0.681 


2 


0.791 


3 


# 


Estonia 


0.672 


3 


0-735 


2 


0.486 


2 


0.598 


3 


# 


France 


0.716 


3 


0.685 


2 


0.544 


2 


0.570 


3 


t,2 


Lithuania 


0.806 


3 


0.805 


2 


0.767 


2 


0-740 


3 


t 


Moscow, Russian 
Federation 


0.711 


3 


0.609 


2 


0.699 


2 


0.671 


3 


# 


Norway 


0.618 


3 


0.782 


2 


0.628 


2 


0-773 


3 


t 


Russian Federation 


0.749 


3 


0.699 


2 


0-737 


2 


0.620 


3 


t 


South Africa 


0.910 


3 


0.911 


2 


0.900 


2 


0.789 


3 


t, 1 


Thailand 


0.856 


3 


0-795 


2 


0-773 


2 


0-739 


3 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 

11.4.5 Impact of ICT Use on Students as Perceived by Teachers 

Question 20 of the teacher questionnaire asked: “To what extent has the use of ICT 
impacted on your students in the target class in the following areas?” Respondents 
were asked to select a response from five choices — “decreased a lot/’ “decreased a 
little” “no impact/’ “increased a little,” and “increased a lot” — placed on a Likert-type 
scale. Responses to this question were used to provide indicators for the ICT impact on 
student orientation. Table 11.7 and Table 11.8 include the reliabilities for the indicators 
on teacher-perceived impact of ICT use on students. In total, there were eight indicators 
in this category, four of which were non-scale indicators comprising only one item: ICT 
skill (20H), learn at own pace (201), achievement gap (20K), and socioeconomic divide 
(200) (one item). The scale indicators were traditionally important (20A and 20N), 
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inquiry skills (20C, 20D, and 20E), collaboration (20F and 20G), and affective impact 
(20B, 20J, 20L, and 20M). 

The reliabilities (Tables 11.7 and 11.8) were all greater than 0.5 for the two 
indicators inquiry skills and collaboration in all participating systems, which was 
statistically acceptable. The reliabilities for traditionally important impact and affective 
impact were greater than 0.5 in most systems. Of the systems that met the participation 
requirement and followed the required administrative procedures, three had reliabilities 
below 0.5 for the affective impact scale. In these three countries — Catalonia, Finland, 
and Slovenia — the reliabilities were above 0.45, except for science teachers in Slovenia. 
The reliabilities were even less satisfactory for the traditionally important impact. Finland, 
Hong Kong SAR, Israel, Japan, and Slovenia had reliabilities below 0.5 for both teacher 
populations. Hence, for further secondary analysis, this indicator should be used only for 
those systems that have acceptable reliabilities. 



Table 11.7: Reliabilities for the Scale Indicators on (Mathematics) Teacher-perceived Impact 
ofICT Use on Students 



Flags 


Education System 


ICT Impact on 
Teacher-Traditionally 
Important 


ICT Impact on 
Student- 
Inquiry Skills 


ICT Impact 
on Student- 
Collaboration 


ICT Impact 
on Student- 
Affective impact 






Cronbach’s 

Alpha 


No . of 
Items 


Cronbach’s 

Alpha 


No . of 
Items 


Cronbach’s 

Alpha 


No . of 
Items 


Cronbach’s 

Alpha 


No . of 
Items 




Catalonia, Spain 


0-553 


2 


0.683 


3 


0.768 


2 


0.481 


4 


1 


Chile 


o.6gi 


2 


0.824 


3 


0.867 


2 


0.736 


4 




Chinese Taipei 


0.511 


2 


0.762 


3 


0-753 


2 


0.572 


4 


2 


Finland 


0-397 


2 


0.546 


3 


0.504 


2 


0.488 


4 


2 


Hong Kong SAR 


0.467 


2 


0.676 


3 


0.838 


2 


0.589 


4 


4 


Israel 


0.363 


2 


0.799 


3 


0.791 


2 


0.709 


4 


1 


Italy 


0.604 


2 


0.702 


3 


0-773 


2 


0.714 


4 


1-3 


Japan 


0-453 


2 


0.786 


3 


0.744 


2 


0.652 


4 


2 


Ontario, Canada 


0-503 


2 


0.714 


3 


0.718 


2 


0.662 


4 




Singapore 


0.610 


2 


0-795 


3 


0.789 


2 


0.675 


4 




Slovak Republic 


0.601 


2 


0.696 


3 


0.694 


2 


0.538 


4 




Slovenia 


0-475 


2 


0-534 


3 


0.768 


2 


0-450 


4 




International alpha on 
adjudicated systems 


0.624 


2 


0.754 


3 


o. 8 oi 


2 


0.708 


4 



Systems not satisfying participation standard or not adhering to survey administration procedure 



t,2 


Alberta, Canada 


0.595 


2 


0.748 


3 


0.738 


2 


0.665 


4 


# 


Denmark 


0-345 


2 


0.642 


3 


0.549 


2 


0.541 


4 


# 


Estonia 


0.581 


2 


0.727 


3 


0.787 


2 


0.531 


4 


# 


France 


0.583 


2 


0-577 


3 


0.756 


2 


0.580 


4 


t, 2 


Lithuania 


0.599 


2 


0.691 


3 


0.853 


2 


0.615 


4 


t 


Moscow, Russian 
Federation 


0.683 


2 


0.761 


3 


0.865 


2 


0.501 


4 


# 


Norway 


0.504 


2 


0.624 


3 


0.519 


2 


0.341 


4 


t 


Russian Federation 


0.568 


2 


0-743 


3 


0.785 


2 


0-573 


4 


t 


South Africa 


0.846 


2 


0.948 


3 


0.923 


2 


0.919 


4 


t,l 


Thailand 


0.608 


2 


0.895 


3 


0.862 


2 


0.723 


4 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 
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Table 11.8: Reliabilities for the Scale Indicators on (Science) Teacher-perceived Impact ofICT 
Use on Students 



Flags 


Education System 


iCT Impact on 
Teacher-Traditionally 
Important 


ICT Impact on 
Student- 
Inquiry Skills 


ICT Impact 
on Student- 
Collaboration 


ICT Impact 
on Student- 
Affective Impact 






Cron bach’s 
Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 




Catalonia, Spain 


0.585 


2 


0.681 


3 


0.711 


2 


0.406 


4 


1 


Chile 


0.625 


2 


0.790 


3 


0.820 


2 


0.744 


4 




Chinese Taipei 


0.594 


2 


0.764 


3 


0.789 


2 


0.677 


4 


2 


Finland 


0.290 


2 


0.657 


3 


0-573 


2 


0.444 


4 


2 


Hong Kong SAR 


0-373 


2 


0.752 


3 


0.767 


2 


0.599 


4 


4 


Israel 


0.483 


2 


0.778 


3 


0.721 


2 


0.662 


4 


1 


Italy 


0.563 


2 


0.706 


3 


0.764 


2 


0.718 


4 


1-3 


Japan 


0.446 


2 


0.698 


3 


0.592 


2 


0.276 


4 


2 


Ontario, Canada 


0.549 


2 


0.742 


3 


0.748 


2 


0.683 


4 




Singapore 


0.490 


2 


0.766 


3 


0.814 


2 


0.668 


4 




Slovak Republic 


0-543 


2 


0.689 


3 


0-737 


2 


0.538 


4 




Slovenia 


0.459 


2 


0.724 


3 


0-793 


2 


0.344 


4 




International alpha on 
adjudicated systems 


0.587 


2 


0.768 


3 


0.787 


2 


0.693 


4 



Systems not satisfying participation standard or not adhering to survey administration procedure 



t, 2 


Alberta, Canada 


0.451 


2 


0.713 


3 


0-735 


2 


0.644 


4 


# 


Denmark 


0-455 


2 


0.725 


3 


0.664 


2 


0.566 


4 


# 


Estonia 


0.425 


2 


0.601 


3 


0-745 


2 


0-397 


4 


# 


France 


0.609 


2 


0.621 


3 


0.607 


2 


0.564 


4 


t,2 


Lithuania 


0-554 


2 


0-539 


3 


0.804 


2 


0.546 


4 


t 


Moscow, Russian 
Federation 


0.687 


2 


0-739 


3 


0.860 


2 


0.382 


4 


# 


Norway 


0.458 


2 


0.523 


3 


0-505 


2 


0.300 


4 


t 


Russian Federation 


0.605 


2 


0-774 


3 


0.855 


2 


0.501 


4 


t 


South Africa 


0.825 


2 


0.941 


3 


0.926 


2 


0.920 


4 


t, 1 


Thailand 


0.530 


2 


0.800 


3 


0.805 


2 


0.624 


4 



Note: 

Section 1 2.3. 1 in this report provides an explanation of the flags reported in the first column of this table. 

11.4.6 Priority for iCT Use in the Near Future 

Question 22 of the teacher questionnaire asked: “Looking ahead to the coming two 
years, what priority will you give to the use of ICT in enhancing your teaching practice in 
the following areas?” Respondents were asked to select a response from four choices — 
“not at all ” “low priority” “medium priority” and “high priority” — placed on a Likert- 
type scale. Responses to this question were used to provide indicators for pedagogical 
104 orientation of the priority areas in ICT use in the near future, while the items from this 

question were used to form indicators of traditionally important (22A, 22B, and 22C), 
lifelong learning (22D, 22E, 22F, 22G, 22H, and 22L), and connectedness (221, 22J, and 
22K) orientations. 

The reliabilities were relatively high (see Table 1 1.9), all being higher than 0.65, 
except for the traditionally important orientation indicator for science teachers in Hong 
Kong, which is still statistically acceptable at 0.541. 
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Table 11.9: Reliabilities for the Indicators on Priority for ICT Use in the Near Future for both 
Mathematics Teachers and Science Teachers 




Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 

11.4.7 Community of Practice 

Question 25 of the teacher questionnaire asked: “To what extent do the following 
statements about school vision apply to the staff in your school?” Question 26 asked: 
“To what extent do the following statements about teachers’ participation in decision- 
making apply to you?” Question 27 asked: “To what extent do the following statements 
about professional collaboration among teachers apply to you?” And Question 28 asked: 
“To what extent do the following statements about support to teachers apply to you?” 
Respondents were asked to select a response from four choices (the four questions all 
have identical responses) — “not at all,” “a little,” “somewhat,” and “a lot” — placed on a 
Likert-type scale. Responses to these questions were used to provide indicators for the 
community of practice orientation. These indicators were shared vision (25A, 25B, and 
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25C), decisionmaking (26A, 26B, and 16C) , professional collaboration {27A, 27B, 27C, 
and 27D), and support (28A, 28B, and 28C). 

Table 11.10 and Table 11.11 show that the reliabilities for these indicators in all 
participating systems were high at 0.5 or above for mathematics teachers^ except for 
Chinese Taipei for the indicators professional collaboration and support, and also high at 
0.5 or above for science teachers. 



Table 11.10: Reliabilities for the Indicators on Community of Practice (COP), Mathematics 
Teachers 



Flags 


Education System 


COP-Shared 

Vision 


COP- 

Decisionmaking 


COP-Professional 

Collaboration 


COP-Support 






Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 




Catalonia, Spain 


0.846 


3 


0.738 


3 


0.521 


4 


0.561 


3 


1 


Chile 


0.889 


3 


0.767 


3 


0.657 


4 


0.647 


3 




Chinese Taipei 


0.761 


3 


0.711 


3 


0-477 


4 


0.425 


3 


2 


Finland 


0.726 


3 


0745 


3 


0.564 


4 


0.561 


3 


2 


Hong Kong SAR 


0.832 


3 


0.750 


3 


0.671 


4 


0.677 


3 


4 


Israel 


0.829 


3 


0.802 


3 


0.565 


4 


0.720 


3 


1 


Italy 


0.796 


3 


0.707 


3 


0.628 


4 


0.529 


3 


1-3 


Japan 


0.654 


3 


0.674 


3 


0.590 


4 


0.391 


3 


2 


Ontario, Canada 


0.837 


3 


0.731 


3 


0.503 


4 


0.646 


3 




Singapore 


0.854 


3 


0.807 


3 


0.641 


4 


0.707 


3 




Slovak Republic 


0773 


3 


0.723 


3 


0-555 


4 


0.664 


3 




Slovenia 


0775 


3 


0.695 


3 


0-545 


4 


0.542 


3 




International alpha on 
adjudicated systems 


0.814 


3 


0.766 


3 


0.589 


4 


o.6io 


3 



Systems not satisfying participation standard or not adhering to survey administration procedure 



t, 2 


Alberta, Canada 


0.820 


3 


0.821 


3 


0-557 


4 


0.689 


3 


# 


Denmark 


0.750 


3 


0-773 


3 


0-533 


4 


0-540 


3 


# 


Estonia 


0.812 


3 


0.699 


3 


0.511 


4 


0.702 


3 


# 


France 


0-743 


3 


0.611 


3 


0.407 


4 


0-570 


3 


t,2 


Lithuania 


0.814 


3 


0.736 


3 


0.626 


4 


0-737 


3 


t 


Moscow, Russian 
Federation 


0.830 


3 


0.803 


3 


0.611 


4 


0.661 


3 


# 


Norway 


0.776 


3 


0.749 


3 


0.447 


4 


0-477 


3 


t 


Russian Federation 


0.812 


3 


0.768 


3 


0.602 


4 


0.717 


3 


t 


South Africa 


0.856 


3 


0-753 


3 


0.645 


4 


0.675 


3 


t, 1 


Thailand 


0.876 


3 


0.822 


3 


0.576 


4 


0.696 


3 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 
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Table 11.11: Reliabilities for the Indicators on Community of Practice (COP), Science 
Teachers 



Flags 


Education System 


COP-Shared 

Vision 


COP- 

Decisionmaking 


COP-Professional 

Collaboration 


COP-Support 






Cron bach’s 
Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 


Cronbach’s 

Alpha 


No. of 
Items 




Catalonia, Spain 


0.798 


3 


0.727 


3 


0.496 


4 


0.592 


3 


1 


Chile 


0.845 


3 


0757 


3 


0.690 


4 


0.679 


3 




Chinese Taipei 


0797 


3 


0.709 


3 


0.493 


4 


0.549 


3 


2 


Finland 


0733 


3 


0.740 


3 


0.542 


4 


0.604 


3 


2 


Hong Kong SAR 


0.837 


3 


0735 


3 


0.641 


4 


0.605 


3 


4 


Israel 


0.857 


3 


0.808 


3 


0-593 


4 


0.728 


3 


1 


Italy 


0.806 


3 


0.708 


3 


0.626 


4 


0-573 


3 


1-3 


Japan 


0.706 


3 


0.655 


3 


0.624 


4 


0.491 


3 


2 


Ontario, Canada 


0.816 


3 


0757 


3 


0-543 


4 


0.606 


3 




Singapore 


0.821 


3 


0.788 


3 


0.531 


4 


0.695 


3 




Slovak Republic 


0.776 


3 


0.746 


3 


0.565 


4 


0.628 


3 




Slovenia 


0.778 


3 


0.706 


3 


0.563 


4 


0.587 


3 




International alpha on 
adjudicated systems 


0.812 


3 


0.766 


3 


0.576 


4 


0.637 


3 



Systems not satisfying participation standard or not adhering to survey administration procedure 



t,2 


Alberta, Canada 


0.824 


3 


0.814 


3 


0.610 


4 


0.696 


3 


# 


Denmark 


0.771 


3 


0.738 


3 


0-554 


4 


0-493 


3 


# 


Estonia 


0.790 


3 


0-775 


3 


0.576 


4 


0.699 


3 


# 


France 


0.746 


3 


0.626 


3 


0.414 


4 


0.636 


3 


t,2 


Lithuania 


0.822 


3 


0-734 


3 


0.644 


4 


0.701 


3 


t 


Moscow, Russian 
Federation 


0.824 


3 


0.791 


3 


0.652 


4 


0.682 


3 


# 


Norway 


0.790 


3 


0.758 


3 


0.406 


4 


0.560 


3 


t 


Russian Federation 


0.800 


3 


0.760 


3 


0.622 


4 


0.716 


3 


t 


South Africa 


0.882 


3 


0.766 


3 


0.666 


4 


0-733 


3 


t, 1 


Thailand 


0.870 


3 


0.783 


3 


0-559 


4 


0.696 


3 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 



77 . 4.8 Assessment 

Question 15 of the teacher questionnaire asked: “In your teaching of the target class in 
this school year, do you use the following methods of assessing student performance?” 
Respondents were asked to select a response from two: “no” and “yes.” Responses to this 
question were used to provide indicators for the assessment orientation. Three indicators 
were computed, namely, traditionally important {ISA and 15B), product (15C, 15D, and 
15E), and reflection/ collaboration (15F, 15G, and 15H). 

The reliahilities of these indicators are shown in Table 11.12. We can see that the 
reliahilities for these indicators were lower than 0.5 in more than half of the instances. 
The reliahilities were particularly problematic in the case of traditionally important 
assessment, being essentially zero for mathematics in Estonia and science in Finland 
and slightly negative for mathematics in Ontario (Canada), Slovenia, Alberta (Canada), 
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and Denmark. Instead of being constructed on the basis of the nature of the assessment 
methods, the three composite indicators were empirically grounded on psychometric 
measures of scale validity The indicator for traditionally important assessment was 
particularly problematic, probably because, as is evident from Figure 5.11 in the 
international report (Law & Chow, 2008), nearly all teachers reported use of these two 
types of assessment (by virtue of being traditionally important). Hence, there was very 
little variance in these two variables, giving rise to the very low reliabilities observed in 
this indicator. 

Table 11.12: Reliabilities for the Assessment Orientation Indicators for both Mathematics 
Teachers and Science Teachers 




Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 
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Because assessment has both a formative and a summative role in the educational 
process, the kinds of assessment used and the importance given to each by a teacher was 
likely to be guided by policies at the school level and beyond. The use of written tests and 
examinations in assessment is prevalent not only because it is traditionally important 
but also because it is likely to be a matter decided at the national, regional, and/ or school 
level rather than an autonomous decision on the part of the teacher. Given the special 
role of assessment in education, it is understandable that assessment practice indicators 
with high face validity may still not be able to achieve the reliability needed to form a 
good scale. In view of these observations, we advise future studies to pay special care to 
the reporting of assessment practices. We also suggest, for the reasons already given, that 
it maybe preferable to report item-level statistics rather than a composite indicator. 

11.4.9 Teachers’ ICT Competence 

Question 21 of the teacher questionnaire asked: “To what extent are you confident 
in accomplishing the following?” Respondents were asked to select a response from 
four choices — “not at all,” “a little,” “somewhat,” and “a lot” — placed on a Likert-type 
scale. Responses to this question were used to provide indicators for the teacher s ICT 
competence orientation. Two sets of indicators were identified — general ICT competence 
( 2 IA, 21B, 21 c, 21D, 21E, 21F, 21G, and 21H) and pedagogical ICT competence (211, 
21J, 21K, 21L, 21M, 21N, and 21P). 

The reliabilities of these indicators, shown in Table 11.13, were very high for all the 
systems (above 0.8 in all cases) for both indicators and for both populations of teachers. 
This outcome indicates that the two ICT-related teacher-competence indicators form 
strong scale indicators. 

11.4.10 Professional Development 

Question 24 of the teacher questionnaire asked: “Have you participated in any of 
the following professional development activities? If no, would you wish to attend?” 
Respondents were asked to select a response from three choices — “no, I do not wish to 
attend,” “no, I would like to attend if available,” and “yes, I have” — placed on a Likert-type 
scale. Responses to this question were used to provide indicators for the professional 
development orientation. Two sets of indicators — technical (24A, 24B, 24C, 24D, and 
24G) and pedagogical (24E and 24F) — with three sub-categories (no, want to, and yes) 
were identified. 

The reliabilities of these indicators are shown in Table 11.14 and Table 11.15. The 
reliabilities for all the systems were satisfactory (almost all exceeded 0.6), with some 
approaching as high as 0.9 for both indicators and for both mathematics and science 
teachers. This outcome denotes that these three indicators are acceptable as scale 
indicators. 

11.4.11 Obstacles in Using ICT in Teaching 

Question 23 of the teacher questionnaire asked: “Do you experience the following 
obstacles in using IGT in your teaching?” Respondents were asked to select a response 
from “no” and “yes,” placed on a binary scale. Responses to this question were used 
to provide indicators for obstacles to ICT use in teaching. Three sets of indicators were 
identified: school (23A, 23B, 23J, 23K, and 23L), teachers (23G, 23D, 23E, 23H, and 
231 ), and students (23F and 23G). 
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Table 11.13: Reliabilities for the Competence Orientation Indicators for both Mathematics and 
Science Teachers 







General ICT Competence 


Pedagogical ICT Competence 


Flags 


Education System 


Cronbach's Alpha 


No . of 


Cronbach’s Alpha 


No . of 






Math 


Science 


Items 


Math 


Science 


Items 




Catalonia, Spain 


0.903 


0.882 


8 


0.915 


0.916 


1 


1 


Chile 


0.917 


0.899 


8 


0.926 


0.923 


1 




ChineseTaipei 


0.911 


0.911 


8 


0.926 


0.908 


1 


2 


Finland 


0.910 


0.902 


8 


0.915 


0.914 


1 


2 


Hong Kong SAR 


0.916 


0.884 


8 


0.938 


0.921 


1 


4 


Israel 


0.934 


0.916 


8 


0-933 


0-933 


1 


1 


Italy 


0.914 


0.907 


8 


0.924 


0.920 


1 


1-3 


Japan 


0.869 


0.853 


8 


0.907 


0.900 


1 


2 


Ontario, Canada 


0.840 


0.834 


8 


0.899 


0.891 


1 




Singapore 


0.865 


0.857 


8 


0.911 


0.902 


1 




Slovak Republic 


0.923 


0.932 


8 


0.934 


0.932 


1 




Slovenia 


0.900 


0.887 


8 


0.915 


0.902 


1 




International alpha on 
adjudicated systems 


0.905 


0.897 


8 


0.922 


0.918 


1 


Systems not satisfying participation standard 


or not adhering to survey administration procedure 


t,2 


Alberta, Canada 


0.834 


0.865 


8 


0.887 


0.898 


1 


# 


Denmark 


0.878 


0.898 


8 


0.908 


0.911 


1 


# 


Estonia 


0.897 


0.883 


8 


0.916 


0.909 


1 


# 


France 


0.878 


0.861 


8 


0.899 


0.876 


1 


t, 2 


Lithuania 


0.926 


0.934 


8 


0.928 


0.933 


1 


t 


Moscow, Russian 
Federation 


0.940 


0.940 


8 


0.948 


0.942 


1 


# 


Norway 


0.836 


0.869 


8 


0.882 


0.891 


1 


t 


Russian Federation 


0.912 


0.919 


8 


0.922 


0.921 


1 


t 


South Africa 


0.956 


0.956 


8 


0.967 


0.962 


1 


t,l 


Thailand 


0.923 


0.932 


8 


0.945 


0.941 


1 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 



The reliabilities of these indicators are shown in Table 11.16. The reliabilities for 
all the systems for both mathematics and science teachers were generally acceptable^ 
with only a few instances falling below 0.5. We note, however, that the teacher-related 
obstacles had particularly high reliabilities (all above 0.68) among the three indicators 
for obstacles teachers experience when trying to use ICT in their teaching. 
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Table 11.14: Reliabilities for the Technical Professional Development Orientation Indicators for both Mathematics and 
Science Teachers 



Flags 


Education System 


Technical PD- 


•No 


Technical PD-Want to 


Technical PD-' 


Ves 


Cronbach’s Alpha 


No . of 


Cronbach’s Alpha 


No . of 


Cronbach’s Alpha 


No. 0/ 






Math 


Science 


Items 


Math 


Science 


Items 


Math 


Science 


Items 




Catalonia, Spain 


0.696 


0.679 


5 


0.616 


0.660 


5 


0.667 


0.635 


5 


1 


Chile 


0.808 


0774 


5 


0.776 


0.715 


5 


0.743 


0.703 


5 




Chinese Taipei 


0.769 


0.787 


5 


0755 


0-779 


5 


0.770 


0-795 


5 


2 


Finland 


0747 


0.721 


5 


0.685 


0.673 


5 


0.730 


0.700 


5 


2 


Hong Kong SAR 


0.848 


0.840 


5 


0.817 


0-777 


5 


0.846 


0.813 


5 


4 


Israel 


0.792 


0774 


5 


0.758 


0-733 


5 


0.769 


0.700 


5 


1 


Italy 


0739 


0.770 


5 


0.699 


0.712 


5 


0.637 


0.655 


5 


1-3 


Japan 


0.821 


0.804 


5 


0774 


0.732 


5 


0.760 


0-753 


5 


2 


Ontario, Canada 


0.723 


0.702 


5 


0.689 


0.642 


5 


0.732 


0.740 


5 




Singapore 


0.731 


0.738 


5 


0.720 


0.714 


5 


0.740 


0.694 


5 




Slovak Republic 


0.756 


0733 


5 


0.634 


0.622 


5 


0.599 


0.604 


5 




Slovenia 


0.702 


0.694 


5 


0.709 


0.667 


5 


0.704 


0.668 


5 




International alpha on 


0.769 


0.760 


5 


0.736 


0.719 


5 


0.734 


0.718 


5 




adjudicated systems 





















Systems not satisfying participation standard or not adhering to survey administration procedure 



t,2 


Alberta, Canada 


0.762 


0.732 


5 


0.708 


0.668 


5 


0.731 


0-759 


5 


# 


Denmark 


0.723 


0-743 


5 


0.658 


0.648 


5 


0.672 


0.681 


5 


# 


Estonia 


0.666 


0.625 


5 


0.650 


0.585 


5 


0.658 


0-534 


5 


# 


France 


0.706 


0.703 


5 


0.698 


0.651 


5 


0-594 


0.641 


5 


t,2 


Lithuania 


0.708 


0.756 


5 


0.618 


0.640 


5 


0.684 


0.680 


5 


T 


Moscow, Russian 
Federation 


0.756 


0.731 


5 


0.656 


0.627 


5 


0-559 


0.630 


5 


# 


Norway 


0.703 


0.710 


5 


0.679 


0.629 


5 


0.702 


0.693 


5 


T 


Russian Federation 


0.721 


0.763 


5 


0.624 


0.683 


5 


0.506 


0-575 


5 


T 


South Africa 


0.849 


0.867 


5 


0.867 


0.863 


5 


0.850 


0.809 


5 


t,i 


Thailand 


0.912 


0.922 


5 


0.807 


0.813 


5 


0.740 


0.752 


5 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 
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Table 11. IS: Reliabilities for the Pedagogical Professional Development Orientation Indicators 
for both Mathematics and Science Teachers 



Flags 


Education System 


Pedagogical PC 


)-No 


Pedagogical PD-Want to 


Pedagogical PD 


-Yes 


Cronbach’s Alpha 


No. of 


Cronbach’s Alpha 


No. of 


Cronbach’s Alpha 


No. of 






Math 


Science 


Items 


Math 


Science 


Items 


Math 


Science 


Items 




Catalonia, Spain 


0.644 


0.665 


2 


0.591 


0.630 


2 


0.601 


0.587 


2 


1 


Chile 


o.6gi 


0.776 


2 


0.632 


0-754 


2 


0.627 


0.744 


2 




Chinese Taipei 


0.721 


0.584 


2 


0.583 


0.570 


2 


0.578 


0.623 


2 


2 


Finland 


0.662 


0.726 


2 


0.600 


0.672 


2 


0.532 


0.653 


2 


2 


Hong Kong SAR 


0.792 


0.772 


2 


0.758 


0-733 


2 


0.761 


0.768 


2 


4 


Israel 


0.644 


0.712 


2 


0.696 


0.701 


2 


0.700 


0.742 


2 


1 


Italy 


0.550 


0.669 


2 


0.546 


0.567 


2 


0.591 


0.532 


2 


1-3 


Japan 


0.726 


0-733 


2 


0.709 


0.667 


2 


0.694 


0.723 


2 


2 


Ontario, Canada 


0.623 


0.598 


2 


0.622 


0.567 


2 


0.686 


0.650 


2 




Singapore 


0.617 


0.616 


2 


0.691 


0.681 


2 


0.758 


0.631 


2 




Slovak Republic 


0.610 


0.613 


2 


0.485 


0.511 


2 


0.495 


0.478 


2 




Slovenia 


0.620 


0.645 


2 


0.568 


0.611 


2 


0.560 


0.599 


2 




International alpha on 


0.653 


0.674 


2 


0.623 


0.646 


2 


0.637 


o.66i 


2 



adjudicated systems 



Systems not satisfying participation standard or not adhering to survey administration procedure 



t,2 


Alberta, Canada 


0.560 


0.635 


2 


0.611 


0.613 


2 


0-559 


0.681 


2 


# 


Denmark 


0.568 


0.529 


2 


0.630 


0.594 


2 


0.626 


0.671 


2 


# 


Estonia 


0-547 


0.567 


2 


0.646 


0.674 


2 


0.659 


0.685 


2 


# 


France 


0.644 


0.485 


2 


0.626 


0.493 


2 


0.641 


0-577 


2 


t,2 


Lithuania 


0.601 


0.604 


2 


0.566 


0.672 


2 


0.582 


0.726 


2 


t 


Moscow, Russian 
Federation 


0-755 


0.727 


2 


0.676 


0.698 


2 


0.668 


0.742 


2 


# 


Norway 


0.729 


0.771 


2 


0.609 


0.631 


2 


0.594 


0.644 


2 


t 


Russian Federation 


0.615 


0.703 


2 


0.600 


0.700 


2 


0.583 


0.730 


2 


t 


South Africa 


0.825 


0-743 


2 


0.837 


0.833 


2 


0.813 


0.846 


2 


t, 1 


Thailand 


0.941 


0.922 


2 


0.787 


0.783 


2 


0.729 


0.724 


2 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 
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Table 11.16: Reliabilities for the Obstacles Orientation Indicators for both Mathematics and 
Science Teachers 







Obstacles-School 


Obstacles-Teachers 


Obstacles-Students 


Flags 


Education System 


Cronbach’s Alpha 


No . of 


Cronbach’s Alpha 


No . of 


Cronbach’s Alpha 


No . of 






Math 


Science 


Items 


Math 


Science 


Items 


Math 


Science 


Items 




Catalonia, Spain 


0.567 


0-579 


5 


0.752 


0.759 


5 


0.623 


0.610 


2 


1 


Chile 


0.567 


0.612 


5 


0.786 


0.760 


5 


0.507 


0.483 


2 




Chinese Taipei 


0.624 


0.603 


5 


0.770 


0.697 


5 


0.649 


0.541 


2 


2 


Finland 


0.494 


0.520 


5 


0.744 


0.782 


5 


0.430 


0.416 


2 


2 


Hong Kong SAR 


0-757 


0.784 


5 


0-753 


0.765 


5 


0.741 


0.784 


2 


4 


Israel 


0.622 


0.635 


5 


0-754 


0-757 


5 


0.664 


0.650 


2 


1 


Italy 


0.613 


0.582 


5 


0.767 


0.772 


5 


0.606 


0.499 


2 


1-3 


Japan 


0.560 


0.484 


5 


0-777 


0.756 


5 


0.712 


0.663 


2 


2 


Ontario, Canada 


0.617 


0.614 


5 


0.789 


0.790 


5 


0.474 


0.418 


2 




Singapore 


0.664 


0.709 


5 


0.789 


0-745 


5 


0.669 


0.644 


2 




Slovak Republic 


0.499 


0.513 


5 


0.738 


0.751 


5 


0.514 


0-573 


2 




Slovenia 


0.442 


0.482 


5 


0.683 


0.710 


5 


0-557 


0-545 


2 




International alpha on 
adjudicated systems 


0.603 


o.6n 


5 


0.765 


0.760 


5 


0.603 


0.573 


2 



Systems not satisfying participation standard or not adhering to survey administration procedure 



t,2 


Alberta, Canada 


0.588 


0.622 


5 


0.752 


0.764 


5 


0.586 


0-505 


2 


# 


Denmark 


0.581 


0.576 


5 


0.710 


0.758 


5 


0.467 


0-559 


2 


# 


Estonia 


0.546 


0-539 


5 


0-737 


0-757 


5 


0.563 


0.442 


2 


# 


France 


0.446 


0.510 


5 


0.783 


0.742 


5 


0.501 


0-499 


2 


t, 2 


Lithuania 


0.479 


0.516 


5 


0.690 


0.695 


5 


0.591 


0.616 


2 


t 


Moscow, Russian 
Federation 


0.540 


0.571 


5 


0.781 


0.763 


5 


0.636 


0-573 


2 


# 


Norway 


0-473 


0.420 


5 


0.718 


0.729 


5 


0.512 


0.523 


2 


t 


Russian Federation 


0.510 


0.523 


5 


0.700 


0.698 


5 


0.483 


0.514 


2 


t 


South Africa 


0.645 


0.649 


5 


0-759 


0-744 


5 


0.782 


0-745 


2 


t, 1 


Thailand 


0.694 


0.699 


5 


0.787 


0.791 


5 


0.798 


0.725 


2 



Note: 

Section 12.3.1 in this report provides an explanation of the flags reported in the first column of this table. 

11.5 Scale Indicator in the School-level Questionnaires 

As explained in Chapter 1 and further elaborated in Chapter 3, the following conceptual 
domains were addressed at the level of schools: “pedagogical practice,” “vision,” 
“infrastructure,” “staff development,” “support,” and “organization and management.” 

The items that were used for collecting information about each of these domains 
were, for the major part, reported in the main report of SITES 2006 (Law, Pelgrum, & 
Plomp, 2008). As is shown in Chapter 12, this reporting consisted of showing the results 
for single items as well as for composites based on a linear combination of responses from 
multiple items that were hypothesized to address an underlying construct. Evidence for 
the appropriateness of these statistical operations was collected during the field trial as 
well as during the processing of data from the main run. 
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The following composite indicators were based on the school-level data: 

• Pedagogical practice indicator (presence of LLL pedagogy): Section 11.5.1 shows the 
reliabilities for this indicator. 

• Vision indicators: With regard to the vision of school leaders^ three indicators were 
specified in the conceptual framework: vision with regard to lifelong learning 
practices, to connectedness, and to traditional pedagogical practices. Moreover, factor 
analyses led us to conclude that a vision regarding ICT use relative to lifelong learning 
could be identified. This indicator, we determined, reflected the extent to which 
school principals encouraged teachers to use ICT for lifelong learning activities. The 
reliabilities for each of these indicators are shown in Sections 1 1.5.2 to 1 1.5.5. 

• Infrastructure indicators: The infrastructure indicators that were described in the main 
report concerned student:computer ratios, student:internet-computer ratios, and the 
extent to which different types of software are available in schools. The reliabilities of 
each of these indicators are described in Sections 1 1.5.6 and 1 1.5.7. 

• Staff development indicators: Two indicators were constructed for staff development: 
(i) the extent to which teachers were required to acquire knowledge and skills in a 
number of areas; and (ii) the extent to which school leaders gave priority to acquiring 
competencies in the area of change management. The reliabilities are reported in 
Sections 11.5.8 and 11.5.9. 

• Support indicators: Two indicators were created for the extent to which support was 
available for teachers. These were technical support and pedagogical support. The 
reliabilities are shown in Sections 11.5.10 and 11.5.11. 

When combining the results from single items into a composite score, it is 
important to ask to what extent the items are internally consistent. To shed light on 
this question, the following sections contain an overview of the reliabilities (across as 
well as within systems) for each of the composites from the list above, as computed in 
SPSS. In the following sections, we refer to questions from the principal and technology 
coordinator questionnaires. These questionnaires can be found as an online appendix at 
http:/ /www.sites2006.net/ appendix. 

In addition to the reliability coefficients reported for each participating 
education system, a cross-system reliability coefficient called the “international alpha 
on adjudicated systems” was computed on equally weighted data from those systems 
that satisfied the sampling standards. These Cronbach’s alpha values are included in this 
technical documentation to aid not only interested readers but also researchers wishing 
to undertake secondary analysis to assess the reliability and validity of scale indicators 
for cross-system comparisons. Because of potential item bias in those systems where 
non-response might not have been random or completely at random, that is, potentially 
related to the variable of interest, we did not consider in our computation of international 
alpha those systems not satisfying the sampling standard. For the same reason, the 
international report does not include international scale score averages or means. 
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11.5.1 Pedagogical Practice 

The main focus of SITES-Ml, SITES-M2, and SITES 2006 was on pedagogical practices 
in relation to ICT. Because Module 1 consisted of a school-level survey only the research 
consortium deemed it important that Module 1 collect from school principals estimates 
of the pedagogical approaches practiced in their respective schools. Although, in SITES 
2006, these estimates could be collected more directly via teachers, the consortium 
decided to include some of the items from Module 1 again in order to investigate changes 
over time. Question 1 (items A-F) of the principal questionnaire was used (answer 
options: 1 = not at all, 2 = to some extent, 3 = a lot) as the vehicle by which to conduct 
comparisons between SITES-Ml and SITES 2006. 

Table 11.17 shows the reliabilities that were observed across and within countries 
for this set of items. 

Table 11.17: Reliabilities of the Indicator “Presence of Lifelong Learning Practices” 




Note: 

Section 12.2 in this report provides an explanation of the flags reported in the first column of this table. 

As can be inferred from Table 11.17, the reliabilities for the indicator of the 
presence of lifelong learning practices ranged from low to fair only. Because these 
reliabilities were not very high, we advise readers to exercise caution when interpreting 
the results in the main report. We also note that in the final report the changes between 
1998 and 2006 are shown separately for each item so that readers can gain a complete 
picture of changes at the level of individual items. 
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n.5.2 Lifelong Learning Pedagogical Vision 

The items C-G from Question 2 of the principal questionnaire were used to create an 
indicator of the lifelong learning pedagogical vision of school leaders. This indicator was 
expressed in terms of the mean score across these items (values: 1 = strongly disagree, 
2 = disagree, 3 = agree, 4 = strongly agree). 

Table 11.18 contains (per country) the reliability indices for this scale. As can 
be observed, the reliabilities for this indicator were quite good (above .70 for almost all 
education systems). 

Table 11.18: Reliabilities of the Indicator “Lifelong Learning Pedagogical Vision” 




Note: 

Section 12.2 in this report provides an explanation of the flags reported in the first column of this table. 
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77.5.5 Connectedness Pedagogical Vision 

To create an indicator of the school’s vision regarding pedagogical connectedness^ we 
used two items (H and I) from Question 2 of the principal questionnaire. We expressed 
this indicator in terms of the mean score across these items (values: 1 = strongly disagree, 
2 = disagree, 3 = agree, 4 = strongly agree). 

The reliahilities for this indicator are shown in Table 11.19. For most education 
systems, the reliahilities for this indicator were generally quite high, hut in some systems 
they were only moderate. 

Table 11.19: Reliabilities of the Indicator “Connectedness Pedagogical Vision' 




Note: 

Section 12.2 in this report provides an explanation of the flags reported in the first column of this table. 
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n.5.4 Traditional Pedagogical Vision 

Two items — A and B in Question 2 of the principal questionnaire — were available to us 
when creating an indicator of the school’s traditional pedagogical vision. We expressed 
this indicator in terms of the mean score across these items (values: 1 = strongly disagree, 
2 = disagree, 3 = agree, 4 = strongly agree). 

The reliabilities for this indicator are shown in Table 11.20. Although this 
indicator comprised only two items, the reliabilities for most education systems were 
quite satisfactory, but only moderate for the others. 

Table 11.20: Reliabilities of the Indicator “Traditional Pedagogical Vision” 




Note: 

Section 12.2 in this report provides an explanation of the flags reported in the first column of this table. 
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77.5.5 Lifelong Learning ICT Vision 

Here, we were able to create an indicator of an ICT-related vision for lifelong learning 
from items C-F and H from Question 3 of the principal questionnaire (values: 1 = not at 
all, 2 = a little, 3 = somewhat, 4 = a lot). 

We expressed this indicator in terms of the mean score across these items. As is 
evident from Table 11.21, the reliabilities for this indicator were all quite high. 



Table 11.21 : Reliabilities of the Indicator "School Leaders’ Visions on the Importance of ICT 
for Lifelong Learning’’ 



Flags Education System 


Cronbach’s Alpha 


No. of Items 


2,3 Alberta, Canada 


0.806 


5 


Catalonia, Spain 


0.834 


5 


1 Chile 


0.893 


5 


ChineseTaipei 


0.856 


5 


2 Finland 


0.767 


5 


2 Hong Kong SAR 


0.848 


5 


4 Israel 


0.881 


5 


1 Italy 


0.852 


5 


1 Japan 


0.806 


5 


2 Lithuania 


0.819 


5 


Moscow, Russian Federation 


0.780 


5 


2 Ontario, Canada 


0.864 


5 


Russian Federation 


0.792 


5 


Singapore 


0.847 


5 


Slovak Republic 


0.724 


5 


Slovenia 


0.752 


5 


South Africa 


0-975 


5 


1 Thailand 


0.854 


5 


International alpha on adjudicated systems 


0.881 


5 


Systems not satisfying sampiing standards 


# Denmark 


0.787 


5 


# Estonia 


0.805 


5 


# France 


0.892 


5 


# Norway 


0.785 


5 



Note: 

Section 12.2 in this report provides an explanation of the flags reported in the first column of this table. 
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77.5.6 Student-.Computer Ratio and StudentAnternet-computer Ratio 

These indicators consisted of a ratio of two variables^ respectively the number of students 
divided by the number of computers available to students at the grade range targeted in 
SITES 2006. We found that it was not possible to calculate a reliability index for these 
indicators, although it appeared from earlier use of them (in SITES-Ml) that the results 
would b e quite plausible when compared with other surveys measuring student : computer 
ratios. 

77.5.7 Software Avaiiabiiity 

We calculated an indicator of infrastructure-software availability by counting the number 
of items (total =13) that the technical questionnaire respondents checked (on Question 
4 ) as “available.” Our next step was to calculate the percentage of the total number of 
items in this question. 

For most education systems, the reliabilities for this indicator of infrastructure- 
software availability were relatively satisfactory, as is shown in Table 11.22. They 
nonetheless varied across education systems, ranging from moderate (e.g., Singapore) to 
high (e.g., Thailand and South Africa). 

Table 11.22: Reliabilities of the Indicator “Software Availability” 




Note: 

Section 12.2 in this report provides an explanation of the flags reported in the first column of this table. 
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77.5.S Teacher Training Requirements 

This indicator was based on items A-J of Question 12 of the principal questionnaire 
(values: 1 = no; 2 = yes, encouraged; 3 = yes^ required). 

The indicator was calculated as the percentage of items for which the answer 
“yes, required” was given. Table 11.23 shows the reliabilities for this indicator. As is 
evident, these were high for all education systems, except Italy. Here, the reliability was 
moderate. 

Table 11.23: Reliabilities of the Indicator "Training Requirements for Teachers" 




Note: 

Section 12.2 in this report provides an explanation of the flags reported in the first column of this table. 
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n.5.9 Leadership Development Priorities 

This indicator consisted of the number of items (from a total of 10) in Question 13 from 
the principal questionnaire that respondents checked as “high priority” divided by the 
total number of items. We then converted this proportion to a scale running from 0 to 
100 . 

In Table 11.24, we can observe that the reliabilities for this indicator (based on 
items recoded to 1 = high priority and 0 = any other scale values) were high in almost all 
education systems. 

Table 1 1 .24: Reliabilities of the Indicator “Leadership Development Priorities” 




Note: 

Section 12.2 in this report provides an explanation of the flags reported in the first column of this table. 
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11.5.10 Technical Support 

We used Question 16 (items A-K) from the technology coordinator questionnaire to 
create this indicator^ which we calculated after first recoding the answer category “not 
applicable” to “missing.” This indicator was expressed in terms of the mean score across 
these items (values: 1 = no support, 2 = some support, 3 = extensive support). 

As is shown in Table 1 1.25, the reliabilities for the indicator of technical support 
were very high in all systems. 



Table 11.25: Reliabilities of the Indicator "Technical Support” 



Flags 


Education System 


Cronbach’s Alpha 


No. of Items 


2,3 


Alberta, Canada 


0.939 


n 




Catalonia, Spain 


0.943 


n 


1 


Chile 


0.959 


n 




ChineseTaipei 


0.928 


11 


2 


Finland 


0.906 


11 


2 


Hong Kong SAR 


0.911 


11 


4 


Israel 


0.948 


11 


1 


Italy 


0.923 


11 


1 


Japan 


0-955 


11 


2 


Lithuania 


0.890 


11 




Moscow, Russian Federation 


0.876 


11 


2 


Ontario, Canada 


0.941 


11 




Russian Federation 


0.868 


11 




Singapore 


0.905 


11 




Slovak Republic 


0.858 


11 




Slovenia 


0.858 


11 




South Africa 


0.959 


11 


1 


Thailand 


0.950 


11 




International alpha on adjudicated systems 


0.936 


11 


Systems not satisfying sampiing standards 


# 


Denmark 


0.895 


11 


# 


Estonia 


0.849 


11 


# 


France 


0-935 


11 


# 


Norway 


0.902 


11 



Note: 

Section 1 2.2 in this report provides an explanation of the flags reported in the first column of this table. 
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n.5.11 Pedagogical Support 

We based this indicator on items A-F from principal questionnaire Question 15, and 
calculated it after first recoding the answer category “not applicable” to “missing.” We 
expressed this indicator in terms of the mean score across these items (values: 1 = not at 
all, 2 = a little, 3 = somewhat, 4 = a lot). 

As is evident from Table 11.26, the reliabilities for this indicator of pedagogical 
support were high in all education systems. 

Table 11.26: Reliabilities of the Indicator "Pedagogical Support” 




Note: 

Section 12.2 in this report provides an explanation of the flags reported in the first column of this table. 
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12. Reporting Data and Indicators 

Willem Johan Pelgrum 
Nancy Law 



12.1 Overview 

The data from SITES 2006 were reported by Law, Pelgrum, and Plomp (2008). Their 
report contains the following chapters: 

• Chapter 1 offers a description of the background of the study, summaries of SITES- 
M1 and SITES-M2, and the participating education systems. 

• Chapter 2 contains a description of the conceptual framework and the research 
questions addressed in this study. The study design and methodology are also 
summarized in this chapter. 

• The next four chapters (3-6) deal respectively with the findings of the study at macro, 
meso, and micro levels and contain information about (respectively) the system- 
level context (Chapter 3), school-level indicators of conditions affecting the use of 
ICT and pedagogy (Chapter 4), and teacher-level indicators regarding pedagogical 
approaches, the use of ICT, and the way these are affected by teacher characteristics, 
including teachers’ perceptions of pertinent school-level conditions (Chapters 5 and 
6 ). 

• Chapter 7 contains information about a part of the teacher questionnaire that was 
included as an international option in SITES 2006. This component consisted of 
teachers’ descriptions of their most satisfying experience with ICT use in teaching 
and the way they perceived the impact of that practice. 

• Chapter 8 focuses on exploring relationships between school- and teacher-level 
indicators to determine if there is evidence that some key strategic factors commonly 
found in ICT-related educational policies do have an impact on teachers’ pedagogical 
use of ICT. 

• Chapter 9 contains a summary of the key findings from SITES 2006 and a discussion 
of the policy implications for teachers, school leaders, and policymakers. 

Chapters 4 to 8 were based on data that were collected in SITES 2006 from 
samples of schools and teachers. The sections below offer a description of the technical 
details of the reporting in these chapters and refer to other chapters in this technical 
report that shed further light on additional details. The data in Chapter 3 were collected 
via a questionnaire that was administered to the national research coordinators (NRCs). 
These data were not processed statistically. Because the reporting consisted mainly of 
summarizing the responses to this questionnaire, the reporting of these data were not 
discussed in a separate section of this chapter. 
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12.2 School-level Data 

The school-level data were based on two questionnaires: a questionnaire for school 
principals and a questionnaire for ICT coordinators (see http://www.sites2006.net/ 
appendix and Chapter 4). Three types of displays were used for reporting the school- 
level data: 

• Tables and figures based on single items firom the questionnaires: These item-statistics 
were, in most tables and figures, expressed in terms of a percentage of respondents who 
marked a particular answer option. The way that these percentages were calculated is 
described in the main report. In all tables, jackknife standard errors (SE) were also 
included for each of the item statistics (see Chapter 10). For simple bar charts based 
on single items, the confidence intervals shown were calculated as -t/- 1.96*SE. The 
more complex bar graphs do not contain standard errors; these can be found online 
at http://www.sites2006.net/appendix. This site also gives references to the table/ 
figure numbers in the main report. 

• Tables and figures based on composites: Composites were created from sets of variables 
on the basis of hypothesized constructs and evidence from factor and reliability 
analyses. The reliability indices for these composites are shown in Chapter 1 1. 

• Bivariate scatter diagrams: Most scatter diagrams showed the country positions on 
two indicators that were also separately reported. Although the standard errors for 
the country positions are not shown in the scatter diagrams, readers can find these in 
the univariate tables. 

Throughout the chapter on school-level data, systems are listed in two sections, in 
alphabetical order within section, and flagged as described in Table 12.1. 

Table 12.1: Flags Used in School-level Exhibits in the International Report 



System 


Flags 


Alberta, Canada 


2,3 


Catalonia, Spain 




Chile 


1 


Chinese Taipei 




Finland 


2 


Hong Kong SAR 


2 


Israel 


4 


Italy 


1 


Japan 


1 


Lithuania 


2 


Moscow, Russian Federation 




Ontario, Canada 


2 


Russian Federation 




Singapore 




Slovak Republic 




Slovenia 




South Africa 




Thailand 


1 


Denmark 


# 


Estonia 


# 


France 


# 


Norway 


# 
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The flags in Table 12.1 follow the assignment of participation categories as 
described in Chapter 10. Those systems that failed to reach Participation Categories 1 
though 4 were reported in the second section of the table, below a line. The meaning of 
these flags is explained in notes below the displays: 

# School participation rate after including replacement schools is below 70% 

1 School participation rate before including replacement schools is below 85% 

2 School participation rate after including replacement schools is below 85% 

3 Less than 70% of the school-level questionnaires in the participating schools 
were returned 

4 National-defined population covers less than 90% of the national-desired 
population. 

The final section of Chapter 4 of the final report contains a display of correlations 
that were observed between the main indicators described in this chapter. These 
correlations were based upon aggregated (system-level) statistics from education systems 
that satisfied the sampling criteria ( 18 of the 22 systems). The correlation display shows 
only the correlations that reached statistical significance (p >.45). 

12.3 Teacher-level Data 

The descriptive statistics based on the teacher-level data from SITES 2006 were reported 
in three chapters. Chapters 5 and 6 contained results from the core part of the teacher 
questionnaire, while Chapter 7 was based on an international option part of this 
questionnaire. 

12.3.1 Data from the Core Component of the Teacher Questionnaire 

The following types of display were used to report the teacher-level data in Chapter 5: 

• Figures showing the pedagogical orientations of the mathematics and the science teachers 
displayed next to each other in the form of stacked bar graphs. Five sets of these 
figures were displayed, three for the overall pedagogical orientation as reflected by 
the teachers’ espoused curriculum goals, their reported teacher practices, and their 
reported student practices, and two for the reported ICT-using teacher practices and 
reported ICT-using student practices. The pedagogical orientations are core indicators 
designed to form scales. The item-contents of the respective scale composites are 
described in the international report (Law et ah, 2008), while Chapter 11 of this 
technical report provides details regarding the reliabilities of these composites. 

• Radar diagrams with three axes representing the traditionally important, lifelong learning, 
and connectedness orientations for comparing teachers’ and students’ practices or for 
comparing overall and ICT-using practices on these indicators. The interpretation of 
these radar diagrams is explained in the text. 

• Tables showing mean values and standard errors for items in a question: The meaning of 
these values is explained below the tables. 

• A table of correlations of system-level means: These correlations were calculated only 
for those education systems that satisfied the participation standard and followed all 
requisite administrative procedures. 

In many figures and tables, the results for mathematics and science teachers 
are shown separately. Because of space and readability considerations, the figures do 
not contain standard errors. These can be found online at http://www.sites2006.net/ 
appendix. 

Chapter 6 contains descriptive statistics of teacher background data (e.g., age, 
gender, qualifications, self-reported competencies) and also indicators that were 
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hypothesized to influence the teaching and learning practices of these teachers. The 
following formats of figures and tables were used in this chapter: 

• Univariate statistics (including standard errors X presented in tables, per education system, 
for mathematics teachers and for science teachers. Due to space considerations^ 
some univariate statistics were presented only for one population of teachers. The 
corresponding tables for the other teacher population can be found at http://www. 
sites2006.net/ appendix. 

• Scatter diagrams showing the position of education systems on two indicators. 

• Stacked bar graphs showing the percentage of each category of response made by science 
teachers and mathematics teachers, and with the graphs displayed next to each other. 

• Results of logistic regression analyses aimed at exploring relationships between the 
different indicators. The technical details of these analyses are explained in the main 
report. 

In general, the chapters on teacher-level data list the participating systems in two 
separate sections, in alphabetical order within each section and flagged as described in 
Table 12.2. 



Table 12.2: Flags Used in Teacher-level Exhibits in the International Report 



System 


Flags 


Catalonia, Spain 




Chile 


1 


Chinese Taipei 




Finland 


2 


Hong Kong SAR 


2 


Israel 


4 


Italy 


1 


Japan 


1-3 


Ontario, Canada 


2 


Singapore 




Slovak Republic 




Slovenia 




Alberta, Canada 


t,2 


Denmark 


# 


Estonia 


# 


France 


# 


Lithuania 


t, 2 


Moscow, Russian Federation 


t 


Norway 


# 


Russian Federation 


t 


South Africa 


t 


Thailand 


t,l 



The flags in Table 12.2 follow the assignment of participation categories as 
described in Chapter 10. Those systems that either failed to reach Participation Categories 
1 through 4 or did not follow the international sampling procedures for target classes 
were reported in the second section of the table, below a line. The meaning of these flags 
is explained as notes below the displays: 
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# School participation rate after including replacement schools is below 70% 

t International procedures for target-class selection was not followed in all 
schools 

1 School participation rate before including replacement schools is below 85% 

2 School participation rate after including replacement schools is below 85% 

3 Teacher-participation data were collected after survey administration 

4 National-defined population covers less than 90% of the national-desired 
population. 

Some of the exhibits in the international report (Figures 6. 1, 6.3^ 6.4^ 6.6, and 6.7) 
pertain to data collected in Part VII of the teacher questionnaire (“Information about 
You and Your School”). Questions in this part were not asked in the context of a specific 
target class. Although the systems are listed in two separate sections in alphabetical order 
within each section of these figures, they are flagged differently, as described in Table 
12.3. 



Table 12.3: Flags Used in Teacher-level Exhibits in the International Report without Target 
Class Reference 



System 


Flags 


Alberta, Canada 


2 


Catalonia, Spain 




Chile 


1 


Chinese Taipei 




Finland 


2 


Hong Kong SAR 


2 


Israel 


4 


Italy 


1 


Japan 


1.3 


Lithuania 


2 


Moscow, Russian Federation 




Ontario, Canada 


2 


Russian Federation 




Singapore 




Slovak Republic 




Slovenia 




South Africa 




Thailand 


1 


Denmark 


# 


Estonia 


# 


France 


# 


Norway 


# 



The flags in Table 12.3 follow the assignment of participation categories as 
described in Chapter 10. Those systems that failed to reach Participation Categories 1 
through 4 were reported in the second section of the table, below a line. The meaning of 
these flags is explained as notes below the displays: 
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# School participation rate after including replacement schools is below 70% 

1 School participation rate before including replacement schools is below 85% 

2 School participation rate after including replacement schools is below 85% 

3 Teacher participation data were collected after survey administration 

4 National-defined population covers less than 90% of the national- desired 
population. 

12.3.2 Data from the International Option of the Teacher Questionnaire 

Chapter 7 of the international report contains a description of a part of the teacher 
questionnaire that qualified as an international option, meaning that countries were 
not obliged to include this part of the questionnaire. This international option focused 
on what teachers perceived as their most satisfying experience with ICT; the questions 
asked regarded content and perceived impact. 

The following formats of figures and tables were used in this chapter: 

• Univariate statistics (including standard errors) per education system for mathematics 
teachers and science teachers presented in tables. 

• Bar graphs and stacked bar graphs showing the percentage of each category of response(s) 
made by teachers. 

Because of space considerations, some univariate statistics were presented for 
the mathematics population of teachers only, and some of the figures do not show the 
standard errors of the statistics. These standard errors can be found as part of the online 
appendices available at http://www.sites2006.net/ appendix. 

Chapter 7 of the international report also contains tables with statistics based 
on internationally pooled data (excluding the education systems that did not meet the 
sampling standards). The calculation of the means and standard errors for these statistics 
is explained below. 

Suppose that 6 denotes the indicator for which statistics need to be calculated 
on the pooled data set. Let 6^ be the indicator calculated for education system C, and 
a( 6^) its jackknife standard error as the square root of the jackknife variance a^( 6^). The 
international mean will be: 



recognizing that the covariances are zero given the independence of the education 
systems. So, concretely, once a standard error per education system is calculated, 

• The squared standard error for each system is calculated, 

• These squares are summed (i.e., the sampling variance), 

• The sum is divided by the square of the number of systems, and 

• The square root of the result provides the international standard error. 




n c=i 



where C denotes the education systems. Using the identity 



VAR(A-l-B) = VAR(A)-l-VAR(B)-l-2COV(A,B), 



the standard error of the international mean is: 
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12.4 Exploratory Analyses in Search of Explanations 

Chapter 8 of the international report contains the results of the exploratory analyses that 
were conducted in order to obtain a first impression of the relationships between school- 
and teacher-level indicators. These analyses were based on two techniques: 

1. Correlation analysis of the mean ICT-using teacher practice orientation scores at the 
system level with the corresponding means for some school-level factors; and 

2. Multilevel analysis relating ICT-using lifelong-learning-oriented practices of a teacher 
with the contextual factors at the teacher s school. 

These analyses were carried out only on data from those education systems that 
satisfied the participation standard and adhered to all requisite administrative procedures. 
The international report contains an introductory description of multilevel modeling as 
well as the details of the analysis outputs. 

Reference 
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Appendix B: Characteristics of the National 
Samples 

For each education system participating in SITES 2006, this appendix describes 
population coverage, exclusion categories, stratification variables, and any deviations 
from the general SITES sampling design. 

B.1 Alberta, Canada 

• School-level exclusions consisted of native schools and very small schools (fewer 
than five students in Grade 8) 

• Explicit stratification by school size 

• Implicit stratification by urbanization and type of school authority, for a total of 60 
implicit strata 

• Ninety-eight very large and large schools selected with certainty 
Table B.l : Allocation of School Sample in Alberta, Canada 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Very Large Schools 


41 


0 


1 


32 


0 


0 


8 


Large Schools 


57 


0 


5 


37 


0 


0 


15 


Medium-size Schools 


67 


0 


1 


54 


0 


0 


12 


Small Schools 


79 


0 


4 


62 


2 


0 


11 


Very Small Schools 


156 


3 


io 


106 


5 


0 


32 


Total 


400 


3 


21 


291 


7 


0 


78 



B.2 Catalonia, Spain 

• School-level exclusions consisted of very small schools (fewer than 20 students in 
Grade 8) 

• Explicit stratification by school type and school size 

• Implicit stratification by school district and town size, for a total of 172 implicit 
strata 

Table B.2: Allocation of School Sample in Catalonia, Spain 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Public, Very Large Schools 


35 


0 


1 


33 


0 


0 


1 


Public, Large Schools 


38 


0 


0 


32 


3 


0 


3 


Public, Medium-size Schools 


40 


0 


4 


35 


0 


0 


1 


Public, Small Schools 


44 


0 


3 


38 


2 


0 


1 


Public, Very Small Schools 


56 


0 


5 


46 


4 


0 


1 


Private, Very Large Schools 


28 


0 


1 


25 


1 


0 


1 


Private, Large Schools 


31 


0 


2 


26 


1 


0 


2 


Private, Medium-size Schools 


36 


0 


3 


31 


1 


0 


1 


Private, Small Schools 


38 


0 


4 


32 


1 


0 


1 


Private, Very Small Schools 


54 1 


3 


42 


3 


0 


5 


Total 


400 


1 


26 


340 


i 6 


0 


17 
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B-3 Chile 

• School-level exclusions consisted of very small schools (fewer than 10 students in 
Grade 8) 

• Explicit stratification by school type^ urbanization^ and school size 

• No implicit stratification 

Table B.3: Allocation of School Sample in Chile 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Public, Rural 


11 


0 


0 


5 


3 


1 


2 


Public, Urban, Very Large 
Schools 


43 


0 


3 


37 


3 


0 


0 


Public, Urban, Large Schools 


48 


0 


3 


39 


4 


1 


1 


Public, Urban, Medium-size 
Schools 


54 


0 


3 


43 


5 


0 


3 


Public, Urban, Small Schools 


63 


0 


2 


46 


5 


2 


8 


Public, Urban , Very Small 
Schools 


100 


0 


1 


71 


10 


4 


H 


Semi-public, Rural 


5 


0 


0 


4 


0 


0 


1 


Semi-public, Urban, Very 
Large Schools 


31 


1 


1 


26 


2 


0 


1 


Semi-public, Urban, Large 
Schools 


37 


0 


3 


31 


1 


0 


2 


Semi-public, Urban, 
Medium-size Schools 


41 


0 


0 


39 


2 


0 


0 


Semi-public, Urban, Small 
Schools 


51 


0 


0 


45 


3 


0 


3 


Semi-public, Urban, Very 
Small Schools 


74 


0 


5 


56 


8 


2 


3 


Private, Rural 


2 


0 


0 


1 


1 


0 


0 


Private, Urban, Very Large 
Schools 


7 


0 


0 


6 


1 


0 


0 


Private, Urban, Large 
Schools 


8 


0 


3 


2 


2 


1 


0 


Private, Urban, Small 
Schools 


10 


0 


2 


5 


1 


1 


1 


Private, Urban, Very Small 
Schools 


15 


0 


0 


7 


2 


0 


6 


Total 


600 


1 


26 


463 


53 


12 


45 
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B-4 Chinese Taipei 

• School-level exclusions consisted of highly specific schools (arts, sports, etc.), foreign language- oriented 
schools, special education schools, and very small schools (one or two Grade 8 classes) 

• Explicit stratification by region and school size 

• No implicit stratification 

Table BA: Allocation of School Sample in Chinese Taipei 



Explicit Stratum 


Totai Sampled 


ineiigibie 


Requirements 


1 Participating Schoois 


Non- 




Schoois 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schoois 


North, Very Large Schools 


24 


0 


0 


24 


0 


0 


0 


North, Large Schools 


25 


0 


0 


25 


0 


0 


0 


North, Medium-size Schoois 


28 


0 


0 


26 


1 


0 


1 


North, Smaii Schools 


34 


0 


0 


33 


1 


0 


0 


North, Very Small Schools 


56 


0 


0 


51 


5 


0 


0 


Central, Very Large Schools 


11 


0 


0 


11 


0 


0 


0 


Central, Large Schools 


13 


0 


0 


13 


0 


0 


0 


Central, Medium-size 
Schools 


13 


0 


0 


12 


1 


0 


0 


Central, Small Schools 


16 


0 


0 


16 


0 


0 


0 


Central, Very Small Schools 


27 


0 


0 


25 


2 


0 


0 


South, Very Large Schools 


17 


0 


0 


17 


0 


0 


0 


South, Large Schools 


18 


0 


0 


18 


0 


0 


0 


South, Medium-size Schools 


21 


0 


0 


21 


0 


0 


0 


South, Small Schools 


26 


0 


0 


26 


0 


0 


0 


South, Very Small Schools 


42 


0 


0 


41 


1 


0 


0 


East, Large Schools 


6 


0 


0 


6 


0 


0 


0 


East, Medium-size Schools 


7 


0 


0 


6 


1 


0 


0 


East, Small Schools 


12 


0 


0 


12 


0 


0 


0 


Islands 


4 


0 


0 


4 


0 


0 


0 


Total 


400 


0 


0 


387 


12 


0 


1 
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B -5 Denmark 

• School-level exclusions consisted of special education schools and very small schools 
(fewer than nine students in Grade 8) 

• Explicit stratification by school size 

• No implicit stratification 

Table B.S: Allocation of School Sample in Denmark 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Very Large Schools 


61 


0 


3 


35 


n 


0 


12 


Large Schools 


68 


0 


12 


33 


6 


1 


16 


Medium-size Schools 


74 


0 


4 


40 


2 


1 


27 


Small Schools 


82 


0 


7 


48 


10 


0 


17 


Very Small Schools 


115 


0 


10 


62 


14 


0 


29 


Total 


400 


0 


36 


2i8 


43 


2 


101 



B.6 Estonia 

• School-level exclusions consisted of Waldorf schools, schools with specialized 
curriculum, schools for undisciplined students, and special education schools 

• Explicit stratification by school size 

• No implicit stratification 

Table B.6: Allocation of School Sample in Estonia 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Very Large Schools 


29 


0 


0 


14 


0 


0 


15 


Large Schools 


37 


0 


0 


22 


0 


0 


15 


Medium-size Schools 


51 


0 


1 


26 


0 


0 


24 


Small Schools 


82 


1 


0 


41 


0 


0 


40 


Very Small Schools 


232 


1 


1 


123 


0 


0 


107 


Total 


431 


2 


2 


226 


0 


0 


201 
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B-7 Finland 

• School-level exclusions consisted of schools on Aland and special education schools 

• Explicit stratification by region, urbanization, and school size 

• No implicit stratification 

• Census of Swedish-speaking schools 

Table B.7: Allocation of School Sample in Finland 



Explicit Stratum 


Totai Sampled 


ineiigibie 


Requirements 


1 Participating Schoois 


Non- 




Schoois 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schoois 


South, Rural, (Very) Large 
Schools 


lO 


0 


1 


6 


1 


0 


2 


South, Rural, Medium-size 
Schools 


13 


0 


0 


9 


4 


0 


0 


South, Urban, Very Large 
Schools 


23 


0 


1 


13 


0 


0 


9 


South, Urban, Large Schools 


25 


0 


1 


16 


2 


0 


6 


South, Urban, Medium-size 
Schools 


26 


0 


1 


12 


4 


0 


9 


South, Urban, Small Schools 


30 


0 


1 


16 


4 


0 


9 


South, Urban, Very Small 
Schools 


40 


0 


5 


23 


3 


0 


9 


West, Rural, (Very) Large 
Schools 


8 


0 


1 


6 


0 


0 


1 


West, Rural, Medium-size 
Schools 


9 


0 


0 


7 


1 


0 


1 


West, Rural, (Very) Small 
Schools 


12 


0 


3 


6 


1 


0 


2 


West, Urban, Very Large 
Schools 


10 


0 


1 


9 


0 


0 


0 


West, Urban, Large Schools 


11 


0 


1 


8 


1 


0 


1 


West, Urban, Medium-size 
Schools 


12 


0 


2 


10 


0 


0 


0 


West, Urban, Small Schools 


13 


0 


3 


9 


1 


0 


0 


West, Urban, Very Small 
Schools 


18 


0 


1 


12 


1 


0 


4 


East, Rural, (Very) Large 
Schools 


7 


0 


1 


4 


0 


0 


2 


East, Rural, Medium-size 
Schools 


9 


0 


0 


8 


0 


0 


1 


East, Rural, (Very) Small 
Schools 


12 


0 


1 


10 


1 


0 


0 


East, Urban, (Very) Large 
Schools 


8 


0 


0 


5 


1 


0 


2 


East, Urban, Medium-size 
Schools 


8 


0 


0 


7 


1 


0 


0 


East, Urban, (Very) Small 
Schools 


11 


0 


0 


10 


1 


0 


0 


North, Rural, (Very) Large 
Schools 


7 


0 


1 


5 


1 


0 


0 


North, Rural, Medium-size 
Schools 


7 


0 


1 


6 


0 


0 


0 


North, Rural, (Very) Small 


11 


0 


0 


9 


1 


0 


1 



Schools 
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Table B.7: Allocation of School Sample in Finland (contd.) 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


North, Urban, Very Large 
Schools 


7 


0 


0 


5 


0 


0 


2 


North, Urban, Large Schools 


8 


0 


0 


5 


0 


0 


3 


North, Urban, Small Schools 


9 


0 


0 


7 


0 


0 


2 


North, Urban, Very Small 
Schools 


12 


0 


0 


9 


1 


0 


2 


Swedish-speaking 


44 


0 


2 


29 


0 


0 


13 


Total 


420 


0 


28 


281 


30 


0 


81 



B.8 France 

• School-level exclusions consisted of schools in the overseas territories 

• Explicit stratification by school type and school size 

• No implicit stratification 

Table B.8: Allocation of School Sample in France 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Priority, Very Large Schools 


10 


0 


1 


6 


1 


0 


2 


Priority, Large Schools 


11 


0 


2 


7 


0 


0 


2 


Priority, Medium-size Schools 


12 


0 


4 


3 


3 


0 


2 


Priority, Small Schools 


14 


0 


2 


7 


0 


2 


3 


Priority, Very Small Schools 


17 


0 


4 


8 


2 


0 


3 


Non-priority, Very Large 
Schools 


38 


0 


4 


19 


4 


4 


7 


Non-priority, Large Schools 


41 


0 


4 


22 


3 


0 


12 


Non-priority, Medium-size 
Schools 


45 


0 


5 


26 


2 


0 


12 


Non-priority, Small Schools 


49 


0 


8 


32 


3 


2 


4 


Non-priority, Very Small 
Schools 


68 


0 


9 


38 


6 


2 


13 


Private, Very Large Schools 


13 


0 


0 


8 


1 


0 


4 


Private, Large Schools 


15 


0 


3 


6 


0 


0 


6 


Private, Medium-size Schools 


16 


0 


1 


7 


2 


1 


5 


Private, Small Schools 


19 


0 


0 


7 


2 


2 


8 


Private, Very Small Schools 


32 


0 


4 


10 


2 


0 


16 


Total 


400 


0 


51 


206 


31 


13 


99 
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B.g Hong Kong SAR 

• School-level exclusions consisted of schools that do not follow the local school 
curriculum (such as international schools) and special education schools 

• Explicit stratification by language of instruction, school type, and school size 

• No implicit stratification 

Table B.9: Allocation of School Sample in Hong Kong SAR 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


English, Boys 


14 


0 


0 


11 


2 


0 


1 


English, Girls 


26 


0 


0 


18 


0 


0 


8 


English, Mixed 


59 


0 


3 


38 


3 


0 


15 


Chinese, Boys 


22 


0 


1 


12 


1 


0 


8 


Chinese,Girls 


10 


0 


0 


8 


1 


0 


1 


Chinese, Mixed, Very Large 
Schools 


46 


0 


2 


36 


1 


0 


7 


Chinese, Mixed, Large 
Schools 


49 


0 


0 


34 


1 


0 


14 


Chinese, Mixed, Medium-size 
Schools 


51 


0 


4 


34 


3 


0 


10 


Chinese, Mixed, Small 
Schools 


54 


0 


0 


35 


1 


0 


18 


Chinese, Mixed, Very Small 
Schools 


69 


0 


3 


43 


1 


0 


22 


Total 


400 


0 


13 


269 


14 


0 


104 
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B.10 Israel 

• Independent orthodox schools were not covered hy the national- defined target 
population 

• School-level exclusions consisted of special education schools 

• Explicit stratification by school type^ sector^ and school size 

• Implicit stratification by district and SES^ for a total of 227 strata 

Table B.IO: Allocation of School Sample in Israel 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Public, Jewish, Very Large 
Schools 


30 


0 


0 


29 


0 


0 


1 


Public, Jewish, Large Schools 


33 


0 


0 


33 


0 


0 


0 


Public, Jewish, Medium-size 
Schools 


36 


0 


0 


36 


0 


0 


0 


Public, Jewish, Small Schools 


39 


1 


1 


36 


1 


0 


0 


Public, Jewish, Very 
Small Schools 


65 


0 


0 


59 


3 


1 


2 


Public, Other Sectors, 
Very Large Schools 


16 


1 


0 


15 


0 


0 


0 


Public, Other Sectors, 
Large Schools 


17 


3 


1 


13 


0 


0 


0 


Public, Other Sectors, 
Medium-size Schools 


18 


2 


1 


15 


0 


0 


0 


Public, Other Sectors, 
Small Schools 


22 


5 


3 


14 


0 


0 


0 


Public, Other Sectors, 
Very Small Schools 


35 


6 


0 


26 


2 


0 


1 


Public, Religious, 
Very Large Schools 


12 


0 


0 


11 


0 


0 


1 


Public, Religious, Large 
Schools 


14 


0 


0 


12 


0 


0 


2 


Public, Religious, 
Medium-size Schools 


16 


0 


4 


12 


0 


0 


0 


Public, Religious, Small 
Schools 


19 


1 


1 


17 


0 


0 


0 


Public, Religious, Very Small 
Schools 


28 


0 


2 


23 


1 


0 


2 


Total 


400 


19 


13 


351 


7 


1 


9 
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B.11 Italy 

• No school-level exclusions 

• Explicit stratification by school size 

• Implicit stratification by region^ for a total of 96 strata 

• In Italy science and mathematics are taught simultaneously as one subject. Half of the 
teachers were asked to fill in the questionnaire with regards to teaching mathematics; 
half of the teachers were asked to do this task with regards to teaching science 



Table B.l 1 : Allocation of School Sample in Italy 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Very Large Schools 


56 


0 


2 


43 


10 


1 


0 


Large Schools 


62 


0 


1 


50 


7 


1 


3 


Medium-size Schools 


70 


0 


7 


52 


8 


1 


2 


Small Schools 


83 


0 


3 


62 


14 


0 


4 


Very Small Schools 


129 


0 


8 


98 


17 


4 


2 


Total 


400 


0 


21 


305 


56 


7 


n 



B.12 Japan 

• School-level exclusions consisted of schools for educable functionally or mentally 
disabled students and very small schools (fewer than nine students in Grade 8) 

• Explicit stratification by school size 

• Implicit stratification by school type and town size (for public), for a total of 15 
implicit strata 

Table B.12: Allocation of School Sample in Japan 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Small Schools 


132 


0 


0 


130 


0 


0 


2 


Medium-size Schools 


144 


0 


0 


142 


0 


0 


2 


Large Schools 


124 


0 


0 


122 


0 


0 


2 


Total 


400 


0 


0 


394 


0 


0 


6 
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B.13 Lithuania 

• School-level exclusions consisted of youth schools, special needs schools, and very 
small schools (fewer than seven students in Grade 8) 

• Explicit stratification by school size 

• No implicit stratification 

Table B.13: Allocation of School Sample in Lithuania 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Very Large Schools 


53 


0 


4 


31 


0 


0 


12 


Large Schools 


56 


0 


9 


45 


0 


0 


2 


Medium-size Schoois 


61 


0 


4 


48 


0 


0 


9 


Smaii Schools 


75 


0 


9 


49 


0 


0 


17 


Very Small Schools 


155 


0 


7 


107 


4 


0 


31 


Total 


400 


0 


33 


286 


4 


0 


77 



B.14 Moscow, Russian Federation 

• School-level exclusions consisted of special education schools 

• Explicit stratification by school category and school size 

• Implicit stratification by school type (for non-advanced), for a total of 15 implicit 
strata 

• To increase precision, those schools from the Russian Federation sample selected 
for the Moscow region were added to the Moscow sample for data analysis (refer 
to Table B.17, explicit strata: Moscow, Adv. Schools, Large Schools; Moscow, Adv. 
Schools, Small Schools; Moscow, Non-adv. Schools, Large Schools; Moscow, Non- 
adv. Schools, Small Schools) 

Table B.14: Allocation of School Sample in Moscow^ Russian Federation 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Adv. Schools, Very Large 
Schools 


9 


0 


0 


9 


0 


0 


0 


Adv. Schools, Large Schools 


10 


0 


0 


10 


0 


0 


0 


Adv. Schools, Medium-size 
Schools 


11 


0 


0 


11 


0 


0 


0 


Adv. Schools, Small Schools 


12 


0 


0 


12 


0 


0 


0 


Adv. Schools, Very Small 
Schools 


15 


0 


0 


15 


0 


0 


0 


Non-adv. Schools, Very Large 
Schools 


53 


0 


0 


53 


0 


0 


0 


Non-adv. Schools, Large 
Schools 


59 


0 


0 


59 


0 


0 


0 


Non-adv. Schools, Medium- 
size Schools 


65 


0 


0 


65 


0 


0 


0 


Non-adv. Schools, Small 
Schools 


72 


0 


0 


72 


0 


0 


0 


Non-adv. Schools, Very Small 
Schools 


94 


0 


0 


94 


0 


0 


0 


Total 


400 


0 


0 


400 


0 


0 


0 
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B.15 Norway 

• School-level exclusions consisted of Sami schools and very small schools (fewer than 
16 students in Grade 8) 

• Explicit stratification by language and school size 

• No implicit stratification 

Table B.IS: Allocation of School Sample in Norway 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Bokmal, Very Large Schools 


46 


0 


1 


33 


1 


0 


11 


Bokmal, Large Schools 


49 


0 


5 


30 


3 


0 


11 


Bokmal, Medium-size 
Schools 


54 


2 


6 


28 


3 


0 


15 


Bokmal, Small Schools 


61 


0 


15 


28 


5 


0 


13 


Bokmal, Very Small Schools 


113 


1 


12 


52 


12 


2 


34 


Nynorsk, Very Large Schools 


10 


0 


2 


7 


0 


0 


1 


Nynorsk, Large Schools 


11 


0 


2 


7 


1 


0 


1 


Nynorsk, Medium-size 
Schools 


13 


0 


2 


6 


1 


0 


4 


Nynorsk, Small Schools 


16 


0 


1 


9 


2 


0 


4 


Nynorsk, Very Small Schools 


27 


0 


2 


11 


3 


0 


11 


Total 


400 


3 


48 


211 


31 


2 


105 



B.16 Ontario, Canada 

• School-level exclusions consisted of private schools, native schools, special education 
schools, and very small schools (fewer than six students in Grade 8) 

• Explicit stratification by language and school size 

• Implicit stratification by school type, for a total of 14 implicit strata 

Table B.16: Allocation of School Sample in Ontario, Canada 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


English, Very Large Schools 


48 


0 


1 


38 


0 


0 


9 


English, Large Schools 


58 


0 


5 


44 


0 


0 


9 


English, Medium-size 
Schools 


71 


0 


2 


63 


0 


0 


6 


English, Small Schools 


83 


0 


5 


69 


0 


0 


9 


English, Very Small Schools 


120 


0 


10 


99 


0 


0 


11 


French, (Very) Large Schools 


7 


0 


0 


6 


0 


0 


1 


French, (Very) Small Schools 


13 


0 


0 


8 


0 


0 


5 


Total 


400 


0 


23 


327 


0 


0 


50 























SITES 2006 TECHNICAL REPORT 



B.17 Russian Federation 

• School-level exclusions consisted of special education schools 

• A PPS sample of 45 regions was selected from a frame of 86 regions. Sixteen regions 
were large enough to he sampled with certainty 

• Explicit stratification by expected ICT usage (in large regions) and school size (in 
large strata) 

• Implicit stratification by location^ for a total of 295 implicit strata 

• To increase precision^ the schools from the Moscow sample were added to the 
Russian Federation sample for data analysis 

Table B.17: Allocation of School Sample in the Russian Federation 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Adygea 


2 


0 


0 


2 


0 


0 


0 


Bashkortostan, Adv. Schools, 
Large 


2 


0 


0 


2 


0 


0 


0 


Bashkortostan, Adv. Schools, 
Small 


4 


0 


0 


4 


0 


0 


0 


Bashkortostan, Non-adv. 
Schools, Large 


5 


0 


0 


5 


0 


0 


0 


Bashkortostan, Non-adv. 
Schools, Small 


10 


0 


0 


10 


0 


0 


0 


Buratia 


5 


0 


0 


5 


0 


0 


0 


Dagestan, Adv. Schools 


2 


0 


0 


2 


0 


0 


0 


Dagestan, Non-adv. Schools, 
Large 


5 


0 


0 


4 


0 


0 


1 


Dagestan, Non-adv. Schools, 
Small 


10 


0 


0 


6 


0 


0 


0 


Karelia, Adv. Schools 


2 


0 


0 


2 


0 


0 


0 


Karelia, Non-adv. Schools 


3 


0 


0 


3 


0 


0 


0 


Komi, Adv. Schools 


2 


0 


0 


2 


0 


0 


0 


Komi, Non-adv. Schools 


3 


0 


0 


3 


0 


0 


0 


Marii Al 


3 


0 


0 


3 


0 


0 


0 


Tatarstan, Adv. Schools, Large 


4 


0 


0 


4 


0 


0 


0 


Tatarstan, Adv. Schools, Small 


6 


0 


0 


6 


0 


0 


0 


Tatarstan, Non-adv. Schools, 
Large 


4 


0 


0 


4 


0 


0 


0 


Tatarstan, Non-adv. Schools, 
Small 


7 


0 


0 


7 


0 


0 


0 


Udmurtia, Adv. Schools 


2 


0 


0 


2 


0 


0 


0 


Udmurtia, Non-adv. Schools 


5 


0 


0 


5 


0 


0 


0 


Altay kr., Adv. Schools 


3 


0 


0 


3 


0 


0 


0 


Altay kr., Non-adv. Schools, 
Large 


3 


0 


0 


3 


0 


0 


0 


Altay kr., Non-adv. Schools, 
Small 


6 


0 


0 


6 


0 


0 


0 


Krasnodar kr., Adv. Schools 


3 


0 


0 


3 


0 


0 


0 
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Table B.l 7 ; Allocation of School Sample in Russian Federation (contd.) 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Krasnodar kr., Non-adv. 
Schools, Large 


5 


0 


0 


5 


o 


0 


0 


Krasnodar kr., Non-adv. 
Schools, Small 


9 


0 


0 


9 


o 


0 


0 


Krasnoyarsk obi., Adv. Schools 


5 


0 


0 


5 


o 


0 


0 


Krasnoyarsk obi., Non-adv. 
Schools, Large 


4 


o 


0 


4 


o 


0 


0 


Krasnoyarsk obL, Non-adv. 
Schools, Small 


7 


o 


o 


7 


o 


0 


0 


Primorsk kr., Adv. Schools 


2 


o 


o 


2 


o 


0 


0 


Primorsk kr., Non-adv. 
Schools, Large 


2 


o 


o 


2 


o 


0 


0 


Primorsk kr., Non-adv. 
Schools, Small 


5 


o 


o 


5 


o 


0 


0 


Stavropol kr., Adv. Schools 


5 


o 


o 


5 


o 


o 


o 


Stavropol kr., Non-adv. 
Schools, Large 


3 


o 


o 


3 


o 


o 


o 


Stavropol kr., Non-adv. 
Schools, Small 


6 


o 


o 


6 


o 


o 


0 


Khabarovsk kr., Adv. Schools 


2 


o 


o 


2 


o 


o 


0 


Khabarovsk kr., Non-adv. 
Schools 


4 


o 


o 


4 


o 


o 


0 


Arhangelsk obi. 


4 


o 


o 


4 


o 


o 


o 


Astrakhan obi. 


4 


0 


o 


4 


o 


o 


o 


Belgorod obL, Adv. Schools 


2 


o 


o 


2 


o 


o 


0 


Belgorod obL, Non-adv. 
Schools 


4 


o 


o 


4 


o 


o 


0 


Bransk obi. 


5 


o 


o 


5 


o 


o 


o 


Vladimir obi. 


5 


o 


o 


5 


o 


o 


0 


Volgograd obL, Adv. Schools 


5 


o 


o 


5 


o 


o 


0 


Volgograd obL, Non-adv. 
Schools, Large 


3 


o 


o 


3 


o 


o 


0 


Volgograd obL, Non-adv. 
Schools, Small 


7 


o 


o 


7 


o 


o 


0 


Voronezh obL, Adv. Schools 


3 


o 


o 


3 


o 


o 


o 


Voronezh obL, Non-adv. 
Schools, Large 


3 


o 


o 


3 


o 


o 


o 


Voronezh obL, Non-adv. 
Schools, Small 


6 


o 


o 


6 


o 


o 


o 


Irkutsk obL, Adv. Schools 


5 


o 


o 


5 


o 


o 


o 


Irkutsk obL, Non-adv. 
Schools, Large 


2 


o 


o 


2 


o 


o 


o 


Irkutsk obL, Non-adv. 


5 


o 


o 


5 


o 


o 


o 



Schools, Small 
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Table B.l 7 ; Allocation of School Sample in Russian Federation (contd.) 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Kaluga obi. 


4 


0 


0 


4 


0 


0 


0 


Kemerovo obi., Adv. Schools 


2 


0 


0 


2 


0 


0 


0 


Kemerovo obi., Non-adv. 
Schools, Large 


4 


0 


0 


4 


0 


0 


0 


Kemerovo obi., Non-adv. 
Schools, Small 


6 


0 


0 


6 


0 


0 


0 


Kirov obi., Adv. Schools 


3 


0 


0 


3 


0 


0 


0 


Kirov obi., Non-adv. Schools 


3 


0 


0 


3 


0 


0 


0 


Kurgan obi. 


4 


0 


0 


4 


0 


0 


0 


Moscow obi., Adv. Schools, 
Large 


5 


0 


0 


5 


0 


0 


0 


Moscow obi., Adv. Schools, 
Small 


7 


0 


0 


7 


0 


0 


0 


Moscow obi., Non-adv. 
Schools, Large 


5 


0 


0 


5 


0 


0 


0 


Moscow obi., Non-adv. 
Schools, Small 


8 


0 


0 


8 


0 


0 


0 


N. Novgorod obi., Adv. Schools 


4 


0 


0 


4 


0 


0 


0 


N. Novgorod obi., Non-adv. 
Schools, Large 


4 


0 


0 


4 


0 


0 


0 


N. Novgorod obi., Non-adv. 
Schools, Small 


8 


0 


0 


8 


0 


0 


0 


Novosibirsk obi., Adv. Schools 


4 


0 


0 


4 


0 


0 


0 


Novosibirsk obi., Non-adv. 
Schools, Large 


4 


0 


0 


4 


0 


0 


0 


Novosibirsk obi., Non-adv. 
Schools, Small 


7 


0 


0 


7 


0 


0 


0 


Orenburg obi., Adv. Schools 


2 


0 


0 


2 


0 


0 


0 


Orenburg obi., Non-adv. 
Schools, Large 


3 


0 


0 


3 


0 


0 


0 


Orenburg obi., Non-adv. 
Schools, Small 


5 


0 


0 


5 


0 


0 


0 


Pensa obi. 


5 


0 


0 


5 


0 


0 


0 


Perm obi., Adv. Schools 


3 


0 


0 


3 


0 


0 


0 


Perm obi., Non-adv. Schools, 
Large 


3 


0 


0 


3 


0 


0 


0 


Perm obi., Non-adv. Schools, 
Small 


7 


0 


0 


7 


0 


0 


0 


Rostov obi., Adv. Schools 


2 


0 


0 


2 


0 


0 


0 


Rostov obi., Non-adv. Schools, 
Large 


4 


0 


0 


4 


0 


0 


0 


Rostov obi., Non-adv. Schools, 
Small 


8 


0 


0 


8 


0 


0 


0 


Razan obi. 


4 


0 


0 


4 


0 


0 


0 
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Table B.l 7 ; Allocation of School Sample in Russian Federation (contd.) 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Samara obi., Adv. Schools, 
Large 


3 


0 


0 


3 


o 


0 


0 


Samara obi., Adv. Schools, 
Small 


3 


0 


0 


3 


o 


o 


o 


Samara obi., Non-adv. 
Schools, Large 


4 


0 


0 


4 


o 


0 


0 


Samara obL, Non-adv. 
Schools, Small 


7 


0 


0 


7 


o 


0 


0 


Saratov obL, Adv. Schools 


2 


0 


o 


2 


o 


0 


0 


Saratov obL, Non-adv. 
Schools, Large 


3 


0 


0 


3 


o 


0 


0 


Saratov obL, Non-adv. 
Schools, Small 


6 


0 


0 


6 


o 


0 


0 


Sakhalin obL, 
Sverdlovsk obL, Adv. 


2 


0 


0 


2 


o 


o 


o 


Schools, Large 


4 


0 


0 


4 


o 


0 


0 


Sverdlovsk obL, Adv. Schools, 
Small 


5 


o 


o 


5 


o 


0 


0 


Sverdlovsk obL, Non-adv. 
Schools, Large 


6 


o 


o 


6 


o 


0 


0 


Sverdlovsk obL, Non-adv. 
Schools, Small 


11 


o 


o 


11 


o 


0 


0 


Tambov obi. 


5 


o 


o 


5 


o 


0 


0 


Tomsk obi. 


4 


o 


o 


4 


o 


0 


0 


Chelyabinsk obL, Large 


4 


o 


o 


4 


o 


0 


0 


Chelyabinsk obL, Small 


8 


o 


o 


8 


o 


0 


0 


Sankt-Peters burg, Adv. 
Schools, Large 


6 


o 


o 


6 


o 


o 


0 


Sankt-Peters burg, Adv. 
Schools, Small 


7 


o 


o 


7 


o 


o 


o 


Sankt-Petersburg, Non-adv. 
Schools, Large 


4 


o 


o 


4 


o 


o 


0 


Sankt-Petersburg, Non-adv. 
Schools, Small 


6 


o 


o 


6 


o 


o 


0 


Hanty-Mansii-ok, Adv. 
Schools 


2 


o 


o 


2 


o 


o 


0 


Hanty-Mansii-ok, Non-adv. 
Schools, Large 


3 


o 


o 


2 


o 


o 


1 


Hanty-Mansii-ok, Non-adv. 
Schools, Small 


4 


o 


o 


4 


o 


o 


0 


Moscow, Adv. Schools, Large 


4 


0 


o 


4 


o 


o 


o 



Schools 
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Table B.l 7; Allocation of School Sample in Russian Federation (contd.) 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Moscow, Adv. Schools, Small 
Schools 


6 


0 


0 


13 


0 


0 


0 


Moscow, Non-adv. Schools, 
Large Schools 


13 


0 


0 


6 


0 


0 


0 


Moscow, Non-adv. Schools, 
Small Schools 


17 


0 


0 


17 


0 


0 


0 


Total 


500 


0 


0 


494 


0 


0 


6 



B.18 Singapore 

• No school-level exclusions 

• Census of schools 

Table B.18: Allocation of School Sample in Singapore 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


I Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Singapore 


164 


0 


0 


164 


0 


0 


0 


Total 


164 


0 


0 


164 


0 


0 


0 



158 
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B.ig Slovak Republic 

• No school-level exclusions 

• Explicit stratification by school type and school size 

• Implicit stratification by region^ for a total of 64 implicit strata 

Table B.l 9: Allocation of School Sample in the Slovak Republic 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Basic, Very Large Schools 


52 


1 


0 


47 


4 


0 


0 


Basic, Large Schools 


58 


2 


0 


51 


4 


0 


1 


Basic, Medium-size Schools 


64 


1 


0 


61 


2 


0 


0 


Basic, Small Schools 


76 


0 


0 


66 


7 


0 


3 


Basic, Very Small Schools 


117 


4 


0 


101 


10 


0 


2 


Gymnasium, Large Schools 


9 


0 


0 


8 


1 


0 


0 


Gymnasium, Medium-size 
Schools 


11 


0 


0 


9 


1 


1 


0 


Gymnasium, Small Schools 


13 


0 


0 


10 


3 


0 


0 


Total 


400 


8 


0 


353 


32 


1 


6 



B.20 Slovenia 

• School-level exclusions consisted of an Italian school and a Waldorf school 

• Explicit stratification by school size 

• No implicit stratification 

Table B.20: Allocation of School Sample in Slovenia 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


Very Large Schools 


38 


0 


1 


36 


0 


0 


1 


Large Schools 


55 


0 


0 


51 


0 


0 


4 


Medium-size Schools 


69 


0 


2 


56 


0 


0 


11 


Small Schools 


91 


1 


1 


78 


0 


0 


n 


Very Small Schools 


168 


0 


1 


160 


0 


0 


7 


Total 


421 


1 


5 


381 


0 


0 


34 



159 
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B.21 South Africa 

• No school-level exclusions 

• Explicit stratification by expected ICT usage, region (for unknown), and school size 

• Implicit stratification by province, for a total of 1 59 implicit strata 

Table B.21 : Allocation of School Sample in South Africa 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


participating 

Schools 


High ICT Usage Expected, 
Very Large Schools 


11 


0 


0 


0 


0 


0 


1 


High ICT Usage Expected, 
Large Schools 


13 


0 


0 


12 


1 


0 


0 


High ICT Usage Expected, 
Medium-size Schools 


14 


0 


0 


14 


0 


0 


0 


High ICT Usage Expected, 
Small Schools 


15 


0 


1 


11 


0 


0 


3 


High ICT Usage Expected, 
Very Small Schools 


24 


0 


0 


22 


0 


0 


2 


Low ICT Usage Expected, 
Very Large Schools 


24 


0 


0 


23 


0 


0 


1 


Low ICT Usage Expected, 
Large Schools 


26 


0 


0 


24 


1 


0 


1 


Low ICT Usage Expected, 
Medium-size Schools 


29 


0 


0 


25 


0 


0 


4 


Low ICT Usage Expected, 
Small Schools 


33 


0 


0 


31 


0 


0 


2 


Low ICT Usage Expected, 
Very Small Schools 


48 


1 


0 


43 


1 


0 


3 


ICT Usage Unknown, 
Gauteng, Western Cape, 
Very Large Schools 


8 


0 


0 


7 


0 


0 


1 


ICT Usage Unknown, 
Gauteng, Western Cape, 
Large Schools 


9 


1 


0 


7 


0 


0 


1 


ICT Usage Unknown, 
Gauteng, Western Cape, 
Medium-size Schools 


10 


0 


1 


6 


0 


0 


3 


ICT Usage Unknown, 
Gauteng, Western Cape, 
Small Schools 


10 


0 


1 


7 


1 


0 


1 


ICT Usage Unknown, 
Gauteng, Western Cape, 
Very Small Schools 


15 


1 


0 


13 


0 


0 


1 


ICT Usage Unknown, 
Other Provinces, 

Very Large Schools 


18 


1 


0 


14 


0 


0 


3 


ICT Usage Unknown, 
Other Provinces, Large 
Schools 


20 


0 


0 


19 


0 


0 


1 
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Table B.21 : Allocation of School Sample in South Africa (contd.) 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


Participating 

Schools 


ICT Usage Unknown, 
Other Provinces, 
Medium-size Schoois 


23 


0 


0 


21 


0 


0 


2 


iCT Usage Unknown, 
Other Provinces, 
Smaii Schools 


28 


0 


0 


24 


1 


0 


3 


ICT Usage Unknown, 
Other Provinces, 

Very Small Schools 


43 


1 


1 


38 


1 


0 


2 


No Computers Available, 
Very Large Schools 


11 


0 


0 


11 


0 


0 


0 


No Computers Available, 
Large Schools 


13 


0 


0 


12 


0 


0 


1 


No Computers Available, 
Medium-size Schools 


14 


0 


0 


13 


0 


0 


1 


No Computers Available, 
Small Schools 


17 


0 


0 


16 


0 


0 


1 


No Computers Available, 
Very Small Schools 


24 


0 


1 


22 


0 


0 


1 


Total 


500 


5 


5 


445 


6 


0 


39 



i6i 
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B.22 Thailand 

• School-level exclusions consisted of demonstration schools of universities, Border 
Patrol Police Bureau schools, Local Administration Department schools, schools in 
provinces with terrorist activity, and very small schools (fewer than nine students in 
Grade 8) 

• Explicit stratification by school type, urbanization, and school size 

• Implicit stratification by groups of Educational Service Areas, for a total of 210 
implicit strata 

Table B.22: Allocation of School Sample in Thailand 



Explicit Stratum 


Total Sampled 


Ineligible 


Requirements 


1 Participating Schools 


Non- 




Schools 


Schools 


Not Met 


Sampled 


First 

Replacement 


Second 

Replacement 


Participating 

Schools 


Public, Urban, Very Large 
Schools 


12 


0 


0 


11 


1 


0 


0 


Public, Urban, Large Schools 


13 


0 


1 


11 


1 


0 


0 


Public, Urban, Medium-size 
Schools 


13 


0 


0 


10 


3 


0 


0 


Public, Urban, Small Schools 


17 


0 


0 


16 


1 


0 


0 


Public, Urban, Very Small 
Schools 


45 


0 


0 


32 


9 


0 


4 


Public, Rural, Very Large 
Schools 


34 


0 


0 


30 


2 


0 


2 


Public, Rural, Large Schools 


38 


0 


2 


35 


1 


0 


0 


Public, Rural, Medium-size 
Schools 


46 


0 


0 


35 


7 


1 


3 


Public, Rural, Small Schools 


65 


0 


1 


55 


8 


0 


1 


Public, Rural, Very Small 
Schools 


117 


0 


0 


89 


13 


2 


13 


Private, Urban, Large Schools 


5 


0 


0 


5 


0 


0 


0 


Private, Urban, Medium-size 
Schools 


7 


0 


0 


6 


0 


0 


1 


Private, Urban, Small Schools 


13 


0 


0 


12 


0 


0 


1 


Private, Rural, Very Large 
Schools 


9 


0 


0 


7 


2 


0 


0 


Private, Rural, Large Schools 


11 


0 


0 


11 


0 


0 


0 


Private, Rural, Medium-size 
Schools 


13 


0 


0 


10 


3 


0 


0 


Private, Rural, Small Schools 


15 


0 


0 


10 


4 


0 


1 


Private, Rural, Very Small 
Schools 


27 


0 


0 


15 


5 


2 


5 


Total 


500 


0 


4 


400 


60 


5 


31 



162 













Appendix C: Sampling Stratification Variables 

This appendix describes the sampling stratification used in national samples (see also 
Appendix B) and the resulting variables in the international database. 

C.1 IDSTRATE (Explicit Stratification) 

The values for the explicit stratification were assigned after completion of the national 
sampling plans and are included in the international database as variable IDSTRATE. 

Stratification codes are comparable only within but not across education systems. 
Specifically school size was used as the final level of explicit stratification wherever 
possible (see Chapter 6). Also, the specific school-size grouping and binning was 
different for each participating system and is therefore not comparable across systems. 
For instance, although the stratification code for “very large schools” in Alberta and 
Denmark was numerically identical, the definition of “very large schools” was different 
for these two systems. 

Table C.l : Explicit Strata and Corresponding Variables Codes 
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Table C.l : Explicit Strata and Corresponding Variables Codes (contd.) 
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Table C.l : Explicit Strata and Corresponding Variables Codes (contd.) 
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Table C.l : Explicit Strata and Corresponding Variables Codes (contd.) 



IDCNTRY 


Education System 


Comment 


0 

00 

m 


Italy 


1 = Very Large Schools 

2 = Large Schools 

3 = Medium-size Schools 

4 = Small Schools 

5 = Very Small Schools 


392 


Japan 


1 = Small Schools 

2 = Medium-size Schools 

3 = Large Schools 


440 


Lithuania 


1 = Very Large Schools 

2 = Large Schools 

3 = Medium-size Schools 

4 = Small Schools 

5 = Very Small Schools 


6431 


Moscow, Russian 
Federation 


1 = Adv. Schools, Very Large Schools 

2 = Adv. Schools, Large Schools 

3 = Adv. Schools, Medium-size Schools 

4 = Adv. Schools, Small Schools 

5 = Adv. Schools, Very Small Schools 

6 = Non-adv. Schools, Very Large Schools 

7 = Non-adv. Schools, Large Schools 

8 = Non-adv. Schools, Medium-size Schools 

9 = Non-adv. Schools, Small Schools 

10 = Non-adv. Schools, Very Small Schools 


578 


Norway 


1 = Bokmal,Very Large Schools 

2 = Bokmal, Large Schools 

3 = Bokmal, Medium-size Schools 

4 = Bokmal, Small Schools 

5 = Bokmal, Very Small Schools 

6 = Nynorsk,Very Large Schools 

7 = Nynorsk, Large Schools 

8 = Nynorsk, Medium-size Schools 

9 = Nynorsk, Small Schools 

10 = Nynorsk, Very Small Schools 


9132 


Ontario, Canada 


1 = English, Very Large Schools 

2 = English, Large Schools 

3 = English, Medium-size Schools 

4 = English, Small Schools 

5 = English, Very Small Schools 

6 = French, (Very) Large Schools 

7 = French, (Very) Small Schools 


643 


Russian Federation 


1 = Adygea 

2 = Bashkortostan, Adv. Schools, Large Schools 

3 = Bashkortostan, Adv. Schools, Small Schools 

4 = Bashkortostan, Non-adv. Schools, Large Schools 

5 = Bashkortostan, Non-adv. Schools, Small Schools 

6 = Buratia 

7 = Dagestan, Adv. Schools 

8 = Dagestan, Non-adv. Schools, Large Schools 

9 = Dagestan, Non-adv. Schools, Small Schools 
10 = Karelia, Adv. Schools 
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Table C.l: Explicit Strata and Corresponding Variables Codes (contd.) 
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Table C.l : Explicit Strata and Corresponding Variables Codes (contd.) 



Education System 



61 = Moscow obi., Adv. Schools, Small Schools 

62 = Moscow obi., Non-adv. Schools, Large Schools 

63 = Moscow obi., Non-adv. Schools, Small Schools 
64= N. Novgorod obi., Adv. Schools 

65 = N. Novgorod obi., Non-adv. Schools, Large Schools 

66 = N. Novgorod obi., Non-adv. Schools, Small Schools 
67= Novosibirsk obi., Adv. Schools 

68 = Novosibirsk obi., Non-adv. Schools, Large Schools 

69 = Novosibirsk obi., Non-adv. Schools, Small Schools 
70= Orenburg obi., Adv. Schools 

71 = Orenburg obi., Non-adv. Schools, Large Schools 

72 = Orenburg obi., Non-adv. Schools, Small Schools 

73 = Pensa obi. 

74= Perm obi., Adv. Schools 

75 = Perm obi., Non-adv. Schools, Large Schools 

76 = Perm obi., Non-adv. Schools, Small Schools 
77= Rostov obi., Adv. Schools 

78 = Rostov obi., Non-adv. Schools, Large Schools 

79 = Rostov obi., Non-adv. Schools, Small Schools 

80 = Razan obi. 

81 = Samara obi., Adv. Schools, Large Schools 

82 = Samara obi., Adv. Schools, Small Schools 

83 = Samara obi., Non-adv. Schools, Large Schools 

84 = Samara obi., Non-adv. Schools, Small Schools 
85= Saratov obi., Adv. Schools 

86 = Saratov obi., Non-adv. Schools, Large Schools 

87 = Saratov obi., Non-adv. Schools, Small Schools 

88 = Sakhalin obi. 

89 = Sverdlovskobl., Adv. Schools, Large Schools 

90 = Sverdlovsk obi., Adv. Schools, Small Schools 

91 = Sverdlovsk obi., Non-adv. Schools, Large Schools 

92 = Sverdlovsk obi., Non-adv. Schools, Small Schools 

93 = Tambov obi. 

94= Tomsk obi. 

95= Chelyabinsk obi.. Large Schools 

96= Chelyabinsk obi.. Small Schools 

101 = Sankt-Petersburg, Adv. Schools, Large Schools 

102= Sankt-Petersburg, Adv. Schools, Small Schools 

103 = Sankt-Petersburg, Non-adv. Schools, Large Schools 

104= Sankt-Petersburg, Non-adv. Schools, Small Schools 

105= Hanty-Mansii-ok, Adv. Schools 

106 = Hanty-Mansii-ok, Non-adv. Schools, Large Schools 

107 = Hanty-Mansii-ok, Non-adv. Schools, Small Schools 

108 = Moscow, Adv. Schools, Very Large Schools 

109 = Moscow, Adv. Schools, Large Schools 

no = Moscow, Adv. Schools, Medium-size Schools 

111 = Moscow, Adv. Schools, Small Schools 

112 = Moscow, Adv. Schools, Very Small Schools 

113 = Moscow, Non-adv. Schools, Very Large Schools 

114 = Moscow, Non-adv. Schools, Large Schools 



Comment 



IDCNTRY 
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Table C.l: Explicit Strata and Corresponding Variables Codes (contd.) 
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Table C.l : Explicit Strata and Corresponding Variables Codes (contd.) 




C.2 IDSTRATU (Implicit Stratification) 

In addition to explicit stratification and sample size allocation, implicit stratification was 
used to order schools on the sampling frame prior to selection (see Chapter 6 for details). 
The SITES 2006 international database features two additional variables — IDSTRATI 
and IDSTRATU — that carry the system-specific information about these stratification 
values. The variables are not labeled in the public data files; readers and database users 
are therefore referred to Table C.2 below. 

IDSTRATI carries information about the implicit stratification based on the 
sampling forms and plans and as assigned during the selection. Values for IDSTRATI 
are conditional on the IDSTRATE values and can only be interpreted in connection 
with that variable. IDSTRATU, however, carries unique information about the implicit 
stratification and takes the same number for each level of IDSTRATE. Secondary 
analysts are therefore encouraged to use the IDSTRATU information in the international 
database for the purpose of analysis and for grouping schools and teachers sharing 
implicit stratification. 



Table C.2: Implicit Strata and Corresponding Variables Codes 



IDCNTRY 


Education System 


Comment 


9134 


Alberta, Canada 


Not available (removed per NRC request) 


724 


Catalonia, Spain 


1 = Area Code BCI: Barcelona -1 (Ciutat), Large Town 






2 = Area Code BCO: Barcelona-ll (Comarques), Large 

Town 

3 = Area Code BCO: Barcelona-ll (Comarques), Medium- 

size Town 

4 = Area Code BCO: Barcelona-ll (Comarques), Small Town 

5 = Area Code BLA: Baix Llobregat-Anoia, Medium-size 

Town 

6 = Area Code BLA: Baix Llobregat-Anoia, Small Town 

7 = Area Code MAR:Maresme, Large Town 
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Table C.2.- Implicit Strata and Corresponding Variables Codes (contd.) 
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Table C.2: Implicit Strata and Corresponding Variables Codes (contd.) 
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Table C.2.- Implicit Strata and Corresponding Variables Codes (contd.) 



IDCNTRY 


Education System 


Comment 


710 


South Africa 


1 = 


Eastern Cape 






2 = 


Free State 






3 = 


Gauteng 






4 = 


KwaZulu Natal 






5 = 


Limpopo 






6 = 


Mpumalanga 






7 = 


Northern Cape 






8 = 


North West 






9 = 


Western Cape 


764 


Thailand 


1 = 


ESA Group 01 






2 = 


ESA Group 02 






3 = 


ESA Group 03 






4 = 


ESA Group 04 






5 = 


ESA Group 05 






6 = 


ESA Group 06 






7 = 


ESA Group 07 






8 = 


ESA Group 08 






9 = 


ESA Group 09 






10 = 


ESAGroup 10 






11 = 


ESAGroup 11 






12 = 


ESAGroup 12 






13 = 


ESAGroup 13 
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Appendix D: Cultural and National 
Adaptations to the Questionnaires 

This appendix describes adaptations to the international versions of the questionnaire 
items made by the national research coordinators (NRCs) during the translation process. 
It provides users with guidance regarding the availability of internationally comparable 
data for use in secondary analyses. 

The adaptations to questionnaires are presented in two sections: (l) common 
cultural adaptations and variables; and (2) other adaptations specific to individual 
systems. 

D.1 Common Cultural Adaptations and Variables 

Cultural adaptations relate to those text passages in the international English version of 
the instruments that had to be adapted (this was mandatory) to the specific national 
settings and terminology. Mandatory cultural adaptations were indicated using 
pointed brackets in the international English version^ for instance^ <target grade>. The 
information included in the tables below details those instances when the version of the 
question administered in an education system differed from the version of the question 
as it appeared in the international version of the questionnaires. 

D.1.1 Grade Range 

References to a particular <grade range> were made in C^uestions 1 and 5 of the technical 
questionnaire. In SITES-M the grade range was generally defined as ranging from target 
grade minus 1 until target grade plus 1. This range was used for questions that were too 
general to ask at the target grade level but for which the research consortium expected 
(sometimes evidence-based) that the answers might differ between^ for instance^ the 
upper- and lower-secondary levels in a school. Special cases consisted of countries where 
a school-level boundary occurred somewhere within this grade range. 

In general, NRCs were advised to use the same translation as in SITES Module 1 
for Population 2, that is, students of age 14 in the eighth month of the school year, and to 
contact the consortium when in doubt. 



Table D.l : Grade Range Adaptation 



IDCNTRY 


Education System 


Adapted Passage (back-translated into English) 


9134 


Alberta, Canada 


Grades 6-9 


724 


Catalonia, Spain 


Grades 7-8 (First cycle of compulsory secondary 
education) 


152 


Chile 


Grades 5-8 


158 


Chinese Taipei 


Grades 7-9 


208 


Denmark 


Grades 7-9 


233 


Estonia 


Grades 7-9 


246 


Finland 


Grades 7-9 


250 


France 


Grades 3-5 


344 


Hong Kong SAR 


Grades 7-9 


376 


Israel 


Grades 7-9 (Grades 6-8 for eight-year schools) 


380 


Italy 


Grades 6-8 



3 See Pelgrum, W. J., & Anderson R. E. (Eds.). (2001). ICT and the emerging paradigm for life-long learning. Amsterdam: International 
Association for the Evaluation of Educational Achievement (pp. 12-13) for details regarding the SITES Module 1 population 
definitions. 
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Table D.l : Grade Range Adaptation (contd.) 



IDCNTRY 


Education System 


Adapted Passage (back-translated into English) 


392 


Japan 


Grades y-g 


440 


Lithuania 


Grades 7-9 


6431 


Moscow, Russian 
Federation 


Grades 5-9 


578 


Norway 


Grades 8-10 


9132 


Ontario, Canada 


Grades 6-8 


643 


Russian Federation 


Grades 5-9 


702 


Singapore 


Grades y-g 


703 


Slovak Republic 


Grades 7-9 


705 


Slovenia 


Grades y-g 


710 


South Africa 


Grades 8-10 


764 


Thailand 


Grades y-g 



D.1.2 Target Grade 

The passage <target grade> was used in the following places: 

• Principal questionnaire: introduction; Questions 2, 3, 4, S, 6, 8, 10^ Ih 12 (and 
instruction before); 14; 15; 17; and 18 

• Technical questionnaire: introduction; questions 3; 4, and 9 



NRCs were asked to consistently replace <target grade> with the grade level that 
was defined in the national sampling plan (see Chapter 6 and Appendix B). 

Table D.2: Target Range Adaptation 



IDCNTRY 


Education System 


Adapted Passage (back-translated into English) 


9134 


Alberta, Canada 


Grade 8 


724 


Catalonia, Spain 


Grade 8 (second year of compulsory secondary education) 


152 


Chile 


Grade 8 


158 


Chinese Taipei 


Grade 8 


208 


Denmark 


Grade 8 


233 


Estonia 


Grade 8 


246 


Finland 


Grade 8 


250 


France 


Grade 4 


344 


Hong Kong SAR 


Grade 8 


376 


Israel 


Grade 8 


0 

00 

m 


Italy 


Grade 8 


392 


Japan 


Grade 8 


440 


Lithuania 


Grade 8 


6431 


Moscow, Russian 
Federation 


Grade 8 


578 


Norway 


Grade 8 


9132 


Ontario, Canada 


Grade 8 


643 


Russian Federation 


Grade 8 


702 


Singapore 


Grade 8 


703 


Slovak Republic 


Grade 8 


705 


Slovenia 


Grade 8 


710 


South Africa 


Grade 8 


764 


Thailand 


Grade 8 
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D.1.3 Language of Instruction 

Question 23 of the principal questionnaire asked for the approximate percentage of 
students in the school who were native speakers of the <national language=language of 
instruction>. Depending on the language version of the questionnaires, the following 
adaptations were used. 



Table D.3: Language of Instruction Adaptation 



IDCNTRY 


Education System 


Adapted Passage (back-translated into English) 


9134 


Alberta, Canada 


English 


724 


Catalonia, Spain 


Catalan 


152 


Chile 


Spanish 


158 


Chinese Taipei 


Chinese 


208 


Denmark 


Danish 


233 


Estonia 


Estonian 


246 


Finland 


Swedish, Finnish 


250 


France 


French 


344 


Hong Kong SAR 


Chinese 


376 


Israel 


Hebrew for Hebrew-speakers and Arab for Arab-speakers 


0 

00 

m 


Italy 


Italian 


392 


Japan 


Japanese 


440 


Lithuania 


Lithuanian 


6431 


Moscow, Russian 
Federation 


Russian 


578 


Norway 


Norwegian 


9132 


Ontario, Canada 


English, French 


643 


Russian Federation 


Russian 


702 


Singapore 


English 


703 


Slovak Republic 


Slovak 


705 


Slovenia 


Slovene 


710 


South Africa 


English 


764 


Thailand 


Thai 



D.1.4 Language of Questionnaire (ITLANC) 

This variable was not part of the questionnaire data itself hut was tracked hy the national 
center. The majority of participating systems used only one language version of the 
questionnaire. The values for the other systems given in Table D.3 indicate the language 
version of the teacher questionnaire that was administered. The corresponding values are 
included in the mathematics and science teacher data files as variable ITLANG. 
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Table DA: Language of Administered Questionnaire Adaptations 



IDCNTRY 


Education System 


Description 


9134 


Alberta, Canada 


1 = English 


724 


Catalonia, Spain 


1 = Catalan 


152 


Chile 


1 = Spanish 


158 


Chinese Taipei 


1 = Traditional Chinese 


208 


Denmark 


1 = Danish 


233 


Estonia 


1 = Estonian 

2 = Russian 


246 


Finland 


1 = Finnish 

2 = Swedish 


250 


France 


1 = French 


344 


Hong Kong SAR 


1 = Traditional Chinese 


376 


Israel 


1 = Hebrew 


0 

00 

m 


Italy 


1 = Italian 


392 


Japan 


1 = Japanese 


440 


Lithuania 


1 = Lithuanian 


6431 


Moscow, Russian 
Federation 


1 = Russian 


578 


Norway 


1 = Norwegian (Bokmal) 


9132 


Ontario, Canada 


1 = English 

2 = French 


643 


Russian Federation 


1 = Russian 


702 


Singapore 


1 = English 


703 


Slovak Republic 


1 = Slovak 


705 


Slovenia 


1 = Slovene 


710 


South Africa 


1 = English 


764 


Thailand 


1 = Thai 



D.2 Education-system-specific Adaptations and Variables 

In line with the rules and guidelines for national adaptations as outlined in Chapter 
5, Table D.4 lists the type of structural adaptation made in the questionnaires and the 
recoding action that was carried out at the lEA Data Processing and Research Center. 

In the documentation^ adaptations have one of two different codes: 

• Code D: National data for a system are included in the international database. 

This code is used for questions where the specific national version was considered 
appropriate for comparison. 

• Code X: National data for a system are not included in the international database. 
This code is used to refer to all questions that were not administered, not applicable, 
or deleted for any of several reasons (e.g., not internationally comparable, removed 
because of NRC request, or removed due to other data problems). 

The column “item” uses the acronym TG to refer to the teacher questionnaire, CT 
to refer to the technical questionnaire, and CP to refer to the principal questionnaire. 







APPENDIX D 



Table D.S: Specific Education System Adaptations 



IDCNTRY 


Education System 


Item 


Adaptation 


Recoding 


Code 


9134 


Alberta, Canada 


TG-33A1 


Category "post-secondary education (e.g., 
teachers college)” (2) was removed from the 
questionnaire because it was not applicable 
to the Alberta context. National categories 
were recoded to fit the international ones. 


NAT-> INT 
1 -> 1 

2 - >3 

3 - >4 


D 


152 


Chile 


TG-03A1 


Because there is no curriculum tracking in 
Chile, the variable was consistently coded to 
"no tracking" (3). 


Code to 3 


D 


152 


Chile 


TG-12A1 


The category "at anytime" (4) was removed 
from the questionnaire because all school 
hours are scheduled. 


n/a 


D 


152 


Chile 


TG-13A1 


The category "at anytime" (4) was removed 
from the questionnaire because all school 
hours are scheduled. 


n/a 


D 


208 


Denmark 


TG-03A1 


Because there is no curriculum tracking in 
Denmark, the variable was consistently coded 
to "no tracking" (3). 


Code to 3 


D 


208 


Denmark 


TG-33A1 


International category 3 "Bachelor's degree" 
was split into two national options as follows: 

1 = Secondary or high school 

2 = Post-secondary education (e.g., teachers' 

college) 

3 = Bachelor's from an institution that is not a 

university 

4 = Bachelor's from a university 

5 = Master's degree or above 

National categories were recoded to fit the 
international ones. 


NAT-> INT 

1 -> 1 

2 -> 2 

3 - >3 

4 - >3 

5 - >4 


D 


233 


Estonia 


TG-03A1 


Because there is no curriculum tracking in 
Estonia, the question was not administered 
and the variable was consistently coded to 
"no tracking." Question 3 was used to collect 
national information about"humanitarian 
tracking” (1), "mathematics-science tracking" (2), 
"no tracking" (3). 


Code to 3 


D 


246 


Finland 


CT-13C1 


Item "head of department” (C) was removed 
from the questionnaire, as it was not applicable 
to the Finnish context. 


n/a 


X 


344 


Hong Kong SAR 


CT-03E1 


Three additional items — G, H, and 1 — were 
added to the questionnaire for national 
purposes. 


n/a 


X 
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Table D.S: Specific Education System Adaptations (contd.) 



IDCNTRY 


Education System 


Item 


Adaptation 


Recoding 


Code 


344 


Hong Kong SAR 


TG-03A1 


The category “no tracking” (3) was removed 
from the questionnaire, as it was not 
appiicabie in the nationai context. 


n/a 


D 


0 

00 

m 


Italy 


CP-21A1 


internationai category “more than 500,000 
peopie" (6) was spiit into nationai 
categories asfoiiows: 

1 = 3,000 peopie or fewer 

2 = 3,001 to 15,000 peopie 

3 = 15,001 to 50,000 peopie 

4 = 50,001 to 100,000 peopie 

5 = 100,001 to 500,000 people 

6 = 500,001 to 1,000,000 people 

7 = More than 1,000,000 people 
National categories were recoded to fit the 
international ones. 


NAT-> INT 

1 -> 1 

2 -> 2 

3 - >3 

4 - >4 

5 - >5 

6- >6 
7->6 


D 


380 


Italy 


CT-03F1 


Item “ICT as separate subject” (F) was not 
applicable for Grades 6-8 in Italy and was 
therefore removed. 


n/a 

n/a 


X 


0 

00 

m 


Italy 


CT-13C1 


Item “head of department” (C) was removed 
from the questionnaire because it is not 
applicable to the Italian context. 


n/a 


X 


0 

00 

m 


Italy 


TG-33A1 


Categories “secondary or high school” (1) and 
“post-secondary education (e.g., teacher's 
college)” (2) were removed because they a 
not applicable to the system in Italy. The 
remaining categories were recoded to fit the 
international ones. 


NAT-> INT 

1- >3 

2 - >4 


D 


0 

00 

m 


Italy 


TG-35A1 


The original no/yes question was expanded 
to cover four types of licenses/certificates in 
Italy as follows: 

1 = Teacher training by a compulsory public 

qualifier competitive examination 

2 = Postgraduate school for secondary school 

teachers'qualification (SSIS - DPR 470/96) 

3 = Special courses for the obtainment of 

teacher training (D.M. n. 21, 2005) 

4 =No teacher training 

National categories were recoded to fit the 
international ones. 


NAT-> INT 

1 -> 2 

2 -> 2 

3 - >2 

4 - >i 


D 


392 


Japan 


CP-05A1-L1 

CP-07A1-K1 

CP-24A1-F1 

CP-26A1-H1 

CP-32A1-J1 

CP-33A1 

CP-34A1-B1 


The international sequence of response 
was changed from “no/yes” to “yes/no.” 


NAT-> INT 
1 -> 2 

2->l 


D 


392 


Japan 


CT-ogAi-Ei 

CT-ioAi-Di 

CT-iiAi-Ji 

CT-13A1-F1 

CT-14A1-F1 

CT-18A1 

CT-19A1-B1 


The international sequence of response 
categories was changed from “no/yes” to 
to “yes/no.” 


NAT-> INT 
1 -> 2 

2->l 


D 
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IDCNTRY 


Education System 


Item 


Adaptation 


Recoding 


Code 


392 


Japan 


TG-ogA2-M2 

TG-14A2-L2 

TG-15A1-H1 

TG-15A2-H2 

TG-16A2-L2 

TG-18A1 

TG-23A1-L1 

TG-29A1 

TG-30A1-B1 

TG-35A1 


The international sequence of response 
categories was changed from "no/yes" to 
"yes/no." 


NAT-> INT 

1 -> 2 

2 -> 1 


D 


578 


Norway 


TG-03A1 


Because there is no curriculum tracking in 
Norway, the variable was consistently coded 
to “no tracking" (3). 


Code to 3 


D 


702 


Singapore 


CP-21A1 


Because Singapore is a city state, the variable 
was consistently coded to "more than 500,000 
people" (6). 


Code to 6 


D 


702 


Singapore 


TG-03A1 


Because secondary schools in Singapore 
currently offer only academic tracks, the variable 
was consistently coded to “academic" (1). 


Code to 1 


D 


702 


Singapore 


TG-37A1 


Removed international option Part VIII. 


Code to “not 
administered" 


X 


702 


Singapore 


TG-38AT 


Removed international option Part VIII. 


Code to “not 
administered" 


X 


702 


Singapore 


TG-39A1-K1 


Removed international option Part VIII. 


Code to “not 
administered" 


X 


702 


Singapore 


TG-40A1-P1 


Removed international option Part VIII. 


Code to “not 
administered" 


X 


702 


Singapore 


TG-41A1-M1 


Removed international option Part VIII. 


Code to “not 
administered" 


X 


703 


Slovak Republic 


CP-31A1 


Due to a conversion error, the first two categories 
were incorrectly displayed in the online version 
of the questionnaire. Data forthis variable were 
therefore not internationally comparable and 
were removed. 


Code to “not 
administered" 


X 


705 


Slovenia 


TG-03A1 


Because there is no curriculum tracking in 
Slovenia, the variable was consistently coded to 
“no tracking" (3). 


Code to 3 


D 


710 


South Africa 


TG-33A1 


International category “Master's degree or above" 
(4) was split into two national options as 
follows: 

1 = Secondary or high school 

2 = Post-secondary education (e.g., teachers' 

college) 

3 = Bachelor's degree 

4 = Honors' degree 

5 = Master's degree or above 

National categories were recoded to fit the 
international ones. 


NAT-> INT 

1 -> 1 

2 -> 2 

3 - >3 

4 - >4 

5 - >4 


D 


764 


Thailand 


CP-20A1-B1 


The maximum grade level in Thailand is Grade 
12, so the last category “13" (14) was removed. 


n/a 


D 










Appendix E: Examples of Teacher Listing and 
Tracking Forms 

This appendix includes the teacher listing forms and the teacher tracking forms used 
for mathematics and for science. The information and IDs that they contain are for a 
fictional school — 9999. 

To illustrate the listing and sampling and administration procedures described in 
Chapters 6 and 1 , especially with respect to the identification of teachers teaching both 
mathematics and science and their corresponding cross-link between the mathematics 
and science listing forms in column (3), the example provided in Section 6.7 has been 
reused here. 

E.1 SITES 2006 (MS): Teacher Listing Form (Mathematics) 

SITES Participant; Sample Country 
School Name: Sample School 

School ID: 9999 

ICT Usage %: []o [x]i-50 [] 51-75 [] 76-100 



( 1 ) 

Mathematics 

ID 


( 2 ) 

Teacher Name 


( 3 ) 

Science 

ID 


( 4 ) 

Year Of 
Birth 


( 5 ) 

Sex 


(6) 

Exclusion 

Status 


( 7 ) 

Courses/Classes 


99990101 


Teacher A 


99990201 


1950 






8a Math, 8b Math 


99990102 


Teacher B 


99990202 


i960 


2 




8c Math 


99990103 


Teacher C 


99990203 


1961 


2 




8d Math 


99990104 


Teacher D 


99990204 


1972 






8e Math 


99990105 


Teacher E 




1953 


2 




8f Math 


99990106 


Teacher E 




1982 






8g Math 


99990107 


Teacher G 




1976 






8h Math 


99990108 


Teacher H 




1979 


2 




81 Math, 8J Math 


99990109 














99990110 














99990111 














99990112 














99990113 














99990114 














99990115 














99990116 















Use additional sheets if necessary 



Sex (Column 5): 1 = female /2 = male / 9 = missing 

Exclusion Status (Column 6): 1 = teaches disabled students only / 2 = nationally defined reason [NRC: Delete if no reason specified] 
/ 9 = missing 

Courses /Classes (Column 7): Name of all the mathematics courses/classes which are taught by the teacher for the target grade. 
Please separate each name with a comma. 
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E.2 sites 2006 (MS): Teacher Listing Form (Science) 



SITES Participant; 
School Name: 
School ID: 

ICT Usage %: 



Sample Country 
Sample School 



9999 

[]o [x]i-50 [] 51-75 [ 176-100 



(1) 

Science 

ID 


(2) 

Teacher Name 


(3) 

Mathematics 

ID 


(4) 

Year of 
Birth 


(5) 

Sex 


( 6 ) 

Exclusion 

Status 


(7) 

Courses/Classes 


99990201 


Teacher A 


99990101 


1950 


1 




8 b Physics, 8 e Physics, 8 a Biology, 
8 e Biology 


99990202 


Teacher B 


99990102 


i960 


2 




8 b Biology, 8 c Biology, 8 d Biology 


99990203 


Teacher C 


99990103 


1961 


2 




8 c Physics, 8 d Physics 


99990204 


Teacher D 


99990104 


1972 


1 




8 a Physics, 8 a Chemistry, 
8 e Chemistry 


99990205 


Teacher 1 




i960 


2 




8 b Chemistry, 8 c Chemistry, 
8 d Chemistry 


99990206 


Teacher J 




1985 


2 




8 e Physics, 8 f Physics 


99990207 


Teacher K 




1989 


2 




8 g Physics, 8 f Chemistry, 
8 g Chemistry 


99990208 


Teacher L 




1968 


1 




8 e Biology, 8 h Chemistry, 
8 i Chemistry 


99990209 














99990210 














99990211 














99990212 














99990213 














99990214 














99990215 














99990216 















Use additional sheets if necessary 

Sex (Column 5): 1 = female / 2 = male / g = missing 

Exclusion Status (Column 6): 1 = teaches disabled students only / 2 = nationally defined reason [NRC: Delete if no reason specified] 
/ 9 = missing 

Courses / Classes (Column 7): Name of all the science courses / classes which are taught by the teacher for the target grade. Please 
separate each name with a comma 



184 



APPENDIX E 



E-3 sites 2006 (MS): Teacher Tracking Form (Mathematics) 



School Name 

Sample School 



SITES Participant 

Sample Country 



[a] 

School ID 


[b] 

Group ID 


[a] 

School ID 


[b] 

Group ID 


9999 


01 


9999 


01 



(1) 


(2) 


(2) 


(3) 


(4) 


(5) 


(6) 


(7) 


(8) 


Teacher 

Name 


Teacher 

ID 


Teacher 

ID 


Check 

Sum 


Exclusion 

Status 


Year of 
Birth 


Sex 


Online 


Participation 

Status 


Teacher A 


99990101 


99990101 


1234 


9 


1950 


1 


X 




Teacher C 


99990103 


99990103 


1234 


9 


1961 


2 


X 




Teacher E 


99990105 


99990105 


1234 


9 


1953 


2 


X 




Teacher G 


99990107 


99990107 


1234 


9 


1976 


1 


X 

















































































































































































































































































































































































































Exclusion Status (Coiumn 4): 1 = teaches disabled students only / 2 = nationally defined reason [NRC: Delete if no reason specified] 
/ 9=missing 

Sex (Coiumn 6):i = female / 2 = male / 9 = missing 

Participation Status (Coiumn 8): NA = left school permanently / R = returned / N = not returned 10 = online 



SITES 2006 TECHNICAL REPORT 



E-4 sites 2006 (MS): Teacher Tracking Form (Science) 



School Name 

Sample School 



SITES Participant 

Sample Country 



[a] 

School ID 


[b] 

Group ID 


[a] 

School ID 


[b] 

Group ID 


9999 


02 


9999 


02 



(1) 


(2) 


(2) 


(3) 


(4) 


(5) 


(6) 


(7) 


(8) 


Teacher 

Name 


Teacher 

ID 


Teacher 

ID 


Check 

Sum 


Exclusion 

Status 


Year of 
Birth 


Sex 


Online 


Participation 

Status 


Teacher B 


99990202 


99990202 


1234 


9 


i960 


2 


X 




Teacher D 


99990204 


99990204 


1234 


9 


1972 


1 


X 




Teacher J 


99990206 


99990206 


1234 


9 


1985 


2 


X 




Teacher L 


99990208 


99990208 


1234 


9 


1968 


1 


X 

















































































































































































































































































































































































































Exclusion Status (Column 4): i=teaches disabled students only / 2 = nationally defined reason [NRC: Delete if no reason specified] 
/ 9 = missing 

Sex (Column 6): 1 = female / 2 = male / 9 = missing 
Participation Status (Column 8): NA = left school permanently 



